xiangfu0 commented on code in PR #18758:
URL: https://github.com/apache/pinot/pull/18758#discussion_r4558888426
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/text/LuceneTextIndexCreator.java:
##########
@@ -153,7 +160,10 @@ public LuceneTextIndexCreator(String column, File
segmentIndexDir, boolean commi
// 1. Not the realtime index, i.e. commit is set to true
// 2. Happens during realtime segment conversion
// 3. Mutable segment not compacted
- _reuseMutableIndex = config.isReuseMutableIndex() && commit &&
realtimeConversion && !mutableSegmentCompacted;
+ // 4. Not a dictionary-based index: the mutable index is per-row, so a
dictionary-based immutable index must be
+ // rebuilt from the sealed dictionary rather than reusing the mutable
index.
+ _reuseMutableIndex = config.isReuseMutableIndex() && commit &&
realtimeConversion && !mutableSegmentCompacted
+ && !_buildOnDictionary;
Review Comment:
Fixed in 9621984b73. Dictionary text mode now retains original dictionary
values during ingestion so the rebuilt index preserves searchable terms after
sealing.
The conversion regression covers native rawValue and legacy
noRawDataForTextIndex with ordinary and compacted sealing. All four replacement
cases failed before the fix, and all six variants now pass. The full focused
run passed 591 tests under JDK 25, plus all four required affected-module
checks and independent code review.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/TextIndexHandler.java:
##########
@@ -336,7 +336,15 @@ private void
createTextIndexForColumn(SegmentDirectory.Writer segmentWriter, Col
_fieldIndexConfigs.get(columnMetadata.getColumnName()),
columnMetadata);
ForwardIndexReaderContext readerContext =
forwardIndexReader.createContext();
TextIndexCreator textIndexCreator =
StandardIndexes.text().createIndexCreator(context, config)) {
- if (columnMetadata.isSingleValue()) {
+ if (config.isBuildOnDictionary()) {
Review Comment:
Fixed in 9621984b73. Dictionary reload branches before temporary
forward-index creation and opens only the dictionary and text creator,
retaining shared sealing, packing, and marker handling.
LuceneTextIndexConfigReloadTest verifies a forward-disabled
dictionary/inverted column, checks the forward index remains absent before
cleanup, and validates queries after combined-format packing. The focused run
passed all 591 tests and required affected-module checks under JDK 25.
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/TextMatchTransformFunction.java:
##########
@@ -106,19 +112,46 @@ public void init(List<TransformFunction> arguments,
Map<String, ColumnContext> c
}
_textIndexReader = indexReader;
+ _columnArg = columnArg;
+ _dictionary = dataSource.getDictionary();
+ _matchingRawValues = null;
}
public int[] transformToIntValuesSV(ValueBlock valueBlock) {
int length = valueBlock.getNumDocs();
initZeroFillingIntValuesSV(length);
- int[] docIds = valueBlock.getDocIds();
- MutableRoaringBitmap indexDocIds = _textIndexReader.isMultiColumn()
- ? ((MultiColumnTextIndexReader) _textIndexReader).getDocIds(_column,
_predicate, _options)
- : _textIndexReader.getDocIds(_predicate, _options);
+ int[] ids;
+ ImmutableRoaringBitmap matchingIds;
+ if (_textIndexReader.isBuildOnDictionary()) {
+ if (_columnArg.getDictionary() == null) {
+ // A standalone dictionary can coexist with a raw forward index.
+ if (_matchingRawValues == null) {
+ matchingIds = _textIndexReader.getDictIds(_predicate, _options);
+ _matchingRawValues = new
ObjectOpenHashSet<>(matchingIds.getCardinality());
+ for (int dictId : matchingIds.toArray()) {
+ _matchingRawValues.add(_dictionary.getStringValue(dictId));
+ }
+ }
+ String[] values = _columnArg.transformToStringValuesSV(valueBlock);
+ for (int i = 0; i < length; i++) {
+ if (_matchingRawValues.contains(values[i])) {
+ _intValuesSV[i] = 1;
+ }
+ }
+ return _intValuesSV;
+ }
+ matchingIds = _textIndexReader.getDictIds(_predicate, _options);
Review Comment:
Fixed in 9621984b73. Matching dictionary IDs are lazily cached per transform
and cleared by init(), including the standalone-dictionary/raw-forward path.
The 10,005-row projection and SUM regression passes. A focused transform
test verifies one search across repeated batches and a new search after init()
for both forward encodings. All 591 focused tests and required affected-module
checks passed under JDK 25.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/text/LuceneTextIndexReader.java:
##########
@@ -214,9 +220,24 @@ private File getTextIndexFile(File segmentIndexDir) {
return file;
}
+ @Override
+ public boolean isBuildOnDictionary() {
+ return _buildOnDictionary;
+ }
+
@Override
public ImmutableRoaringBitmap getDictIds(String searchQuery) {
- throw new UnsupportedOperationException("");
+ return getDictIds(searchQuery, null);
+ }
+
+ @Override
+ public ImmutableRoaringBitmap getDictIds(String searchQuery, @Nullable
String optionsString) {
+ if (_buildOnDictionary) {
+ // The translator resolves Lucene document IDs to the original
dictionary IDs, including after merges.
+ return getDocIds(searchQuery, optionsString, null);
Review Comment:
Fixed in 9621984b73. A backward-compatible getDictIds(query, options, cache)
default overload forwards the evaluator QueryContext to Lucene’s existing
parsed-query cache.
The regression shares one cache across two dictionary indexes with different
ID mappings and verifies a single parse with correct segment-specific matches.
All 591 focused tests and required affected-module checks passed under JDK 25.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]