Jackie-Jiang commented on code in PR #18758:
URL: https://github.com/apache/pinot/pull/18758#discussion_r4595739608
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/text/LuceneTextIndexReader.java:
##########
@@ -214,9 +220,30 @@ private File getTextIndexFile(File segmentIndexDir) {
return file;
}
+ @Override
+ public boolean isBuildOnDictionary() {
+ return _buildOnDictionary;
+ }
+
@Override
public ImmutableRoaringBitmap getDictIds(String searchQuery) {
- throw new UnsupportedOperationException("");
+ return getDictIds(searchQuery, null);
+ }
+
+ @Override
+ public ImmutableRoaringBitmap getDictIds(String searchQuery, @Nullable
String optionsString) {
+ return getDictIds(searchQuery, optionsString, null);
+ }
+
+ @Override
+ public ImmutableRoaringBitmap getDictIds(String searchQuery, @Nullable
String optionsString,
+ @Nullable QueryScopedCache queryScopedCache) {
+ if (_buildOnDictionary) {
+ // The translator resolves Lucene document IDs to the original
dictionary IDs, including after merges.
+ return getDocIds(searchQuery, optionsString, queryScopedCache);
Review Comment:
[P3] Make the reader's ID contract explicit
In dictionary mode, public `getDocIds()` returns dictionary IDs and
`getNumMatchingDocs()` counts distinct values. Current planner callers dispatch
correctly, but the API contract is misleading. Extract the shared search into a
private helper and reject row-ID/count operations in dictionary mode, following
the existing FST reader pattern.
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/TextMatchTransformFunction.java:
##########
@@ -106,19 +114,50 @@ public void init(List<TransformFunction> arguments,
Map<String, ColumnContext> c
}
_textIndexReader = indexReader;
+ _columnArg = columnArg;
+ _dictionary = dataSource.getDictionary();
+ _matchingRawValues = null;
+ _matchingDictIds = null;
}
public int[] transformToIntValuesSV(ValueBlock valueBlock) {
int length = valueBlock.getNumDocs();
initZeroFillingIntValuesSV(length);
- int[] docIds = valueBlock.getDocIds();
- MutableRoaringBitmap indexDocIds = _textIndexReader.isMultiColumn()
- ? ((MultiColumnTextIndexReader) _textIndexReader).getDocIds(_column,
_predicate, _options)
- : _textIndexReader.getDocIds(_predicate, _options);
+ int[] ids;
+ ImmutableRoaringBitmap matchingIds;
+ if (_textIndexReader.isBuildOnDictionary()) {
+ if (_matchingDictIds == null) {
+ _matchingDictIds = _textIndexReader.getDictIds(_predicate, _options);
+ }
+ matchingIds = _matchingDictIds;
+ if (_columnArg.getDictionary() == null) {
+ // A standalone dictionary can coexist with a raw forward index.
+ if (_matchingRawValues == null) {
+ _matchingRawValues = new
ObjectOpenHashSet<>(matchingIds.getCardinality());
Review Comment:
[P3] Short-circuit constant projection results
When no dictionary values match, return the initialized zero array. When
every dictionary value matches, fill the active output with ones. This avoids
reading forward values for every block and avoids constructing the entire
matching-string set for RAW match-all queries. Existing null propagation
remains separate; the predicate evaluator already uses these shortcuts.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/TextIndexHandler.java:
##########
@@ -201,12 +211,40 @@ private Set<String>
getExistingColumns(SegmentDirectory.Reader segmentReader) {
}
private void validateRecoverySource(SegmentDirectory.Reader segmentReader,
String column) {
- if (!segmentReader.hasIndexFor(column, StandardIndexes.forward())) {
+ if
(_fieldIndexConfigs.get(column).getConfig(StandardIndexes.text()).isBuildOnDictionary())
{
+ if (!segmentReader.hasIndexFor(column, StandardIndexes.dictionary())) {
+ throw new IllegalStateException("Cannot rebuild Lucene text index for
column: " + column
+ + ". Original dictionary must be retained; restore a compatible
segment or refresh it from source data");
Review Comment:
[P2] Accept raw values while dictionary creation is pending
Recovery validation runs before `ForwardIndexHandler` creates the
dictionary. Migrating a single-value RAW column to dictionary text mode
therefore fails when its text index is incompatible or has an interruption
marker, despite an intact raw forward index containing the original values.
Allow that recoverable source when dictionary creation is pending, and extend a
RAW rollback test with `buildOnDictionary=true`.
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/TextMatchTransformFunction.java:
##########
@@ -106,19 +114,50 @@ public void init(List<TransformFunction> arguments,
Map<String, ColumnContext> c
}
_textIndexReader = indexReader;
+ _columnArg = columnArg;
+ _dictionary = dataSource.getDictionary();
+ _matchingRawValues = null;
+ _matchingDictIds = null;
}
public int[] transformToIntValuesSV(ValueBlock valueBlock) {
int length = valueBlock.getNumDocs();
initZeroFillingIntValuesSV(length);
- int[] docIds = valueBlock.getDocIds();
- MutableRoaringBitmap indexDocIds = _textIndexReader.isMultiColumn()
- ? ((MultiColumnTextIndexReader) _textIndexReader).getDocIds(_column,
_predicate, _options)
- : _textIndexReader.getDocIds(_predicate, _options);
+ int[] ids;
+ ImmutableRoaringBitmap matchingIds;
+ if (_textIndexReader.isBuildOnDictionary()) {
+ if (_matchingDictIds == null) {
+ _matchingDictIds = _textIndexReader.getDictIds(_predicate, _options);
Review Comment:
[P3] Share parsed queries in projections too
This two-argument call bypasses the query-scoped parser cache, so
`SELECT`/`SUM(TEXT_MATCH(...))` still parses once per segment. Capture the
cache through the existing `QueryContext` initialization overload and pass it
to the three-argument `getDictIds()`, as filtering already does. Keep matching
bitmaps local to each transform and reset the cache reference on
reinitialization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]