xiangfu0 commented on code in PR #18758:
URL: https://github.com/apache/pinot/pull/18758#discussion_r4558888426


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/text/LuceneTextIndexCreator.java:
##########
@@ -153,7 +160,10 @@ public LuceneTextIndexCreator(String column, File 
segmentIndexDir, boolean commi
       // 1. Not the realtime index, i.e. commit is set to true
       // 2. Happens during realtime segment conversion
       // 3. Mutable segment not compacted
-      _reuseMutableIndex = config.isReuseMutableIndex() && commit && 
realtimeConversion && !mutableSegmentCompacted;
+      // 4. Not a dictionary-based index: the mutable index is per-row, so a 
dictionary-based immutable index must be
+      //    rebuilt from the sealed dictionary rather than reusing the mutable 
index.
+      _reuseMutableIndex = config.isReuseMutableIndex() && commit && 
realtimeConversion && !mutableSegmentCompacted
+          && !_buildOnDictionary;

Review Comment:
   Fixed in 9621984b73. Dictionary text mode now retains original dictionary 
values during ingestion so the rebuilt index preserves searchable terms after 
sealing.
   
   The conversion regression covers native rawValue and legacy 
noRawDataForTextIndex with ordinary and compacted sealing. All four replacement 
cases failed before the fix, and all six variants now pass. The full focused 
run passed 591 tests under JDK 25, plus all four required affected-module 
checks and independent code review.



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/TextIndexHandler.java:
##########
@@ -336,7 +336,15 @@ private void 
createTextIndexForColumn(SegmentDirectory.Writer segmentWriter, Col
         _fieldIndexConfigs.get(columnMetadata.getColumnName()), 
columnMetadata);
         ForwardIndexReaderContext readerContext = 
forwardIndexReader.createContext();
         TextIndexCreator textIndexCreator = 
StandardIndexes.text().createIndexCreator(context, config)) {
-      if (columnMetadata.isSingleValue()) {
+      if (config.isBuildOnDictionary()) {

Review Comment:
   Fixed in 9621984b73. Dictionary reload branches before temporary 
forward-index creation and opens only the dictionary and text creator, 
retaining shared sealing, packing, and marker handling.
   
   LuceneTextIndexConfigReloadTest verifies a forward-disabled 
dictionary/inverted column, checks the forward index remains absent before 
cleanup, and validates queries after combined-format packing. The focused run 
passed all 591 tests and required affected-module checks under JDK 25.



##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/TextMatchTransformFunction.java:
##########
@@ -106,19 +112,46 @@ public void init(List<TransformFunction> arguments, 
Map<String, ColumnContext> c
     }
 
     _textIndexReader = indexReader;
+    _columnArg = columnArg;
+    _dictionary = dataSource.getDictionary();
+    _matchingRawValues = null;
   }
 
   public int[] transformToIntValuesSV(ValueBlock valueBlock) {
     int length = valueBlock.getNumDocs();
     initZeroFillingIntValuesSV(length);
 
-    int[] docIds = valueBlock.getDocIds();
-    MutableRoaringBitmap indexDocIds = _textIndexReader.isMultiColumn()
-        ? ((MultiColumnTextIndexReader) _textIndexReader).getDocIds(_column, 
_predicate, _options)
-        : _textIndexReader.getDocIds(_predicate, _options);
+    int[] ids;
+    ImmutableRoaringBitmap matchingIds;
+    if (_textIndexReader.isBuildOnDictionary()) {
+      if (_columnArg.getDictionary() == null) {
+        // A standalone dictionary can coexist with a raw forward index.
+        if (_matchingRawValues == null) {
+          matchingIds = _textIndexReader.getDictIds(_predicate, _options);
+          _matchingRawValues = new 
ObjectOpenHashSet<>(matchingIds.getCardinality());
+          for (int dictId : matchingIds.toArray()) {
+            _matchingRawValues.add(_dictionary.getStringValue(dictId));
+          }
+        }
+        String[] values = _columnArg.transformToStringValuesSV(valueBlock);
+        for (int i = 0; i < length; i++) {
+          if (_matchingRawValues.contains(values[i])) {
+            _intValuesSV[i] = 1;
+          }
+        }
+        return _intValuesSV;
+      }
+      matchingIds = _textIndexReader.getDictIds(_predicate, _options);

Review Comment:
   Fixed in 9621984b73. Matching dictionary IDs are lazily cached per transform 
and cleared by init(), including the standalone-dictionary/raw-forward path.
   
   The 10,005-row projection and SUM regression passes. A focused transform 
test verifies one search across repeated batches and a new search after init() 
for both forward encodings. All 591 focused tests and required affected-module 
checks passed under JDK 25.



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/text/LuceneTextIndexReader.java:
##########
@@ -214,9 +220,24 @@ private File getTextIndexFile(File segmentIndexDir) {
     return file;
   }
 
+  @Override
+  public boolean isBuildOnDictionary() {
+    return _buildOnDictionary;
+  }
+
   @Override
   public ImmutableRoaringBitmap getDictIds(String searchQuery) {
-    throw new UnsupportedOperationException("");
+    return getDictIds(searchQuery, null);
+  }
+
+  @Override
+  public ImmutableRoaringBitmap getDictIds(String searchQuery, @Nullable 
String optionsString) {
+    if (_buildOnDictionary) {
+      // The translator resolves Lucene document IDs to the original 
dictionary IDs, including after merges.
+      return getDocIds(searchQuery, optionsString, null);

Review Comment:
   Fixed in 9621984b73. A backward-compatible getDictIds(query, options, cache) 
default overload forwards the evaluator QueryContext to Lucene’s existing 
parsed-query cache.
   
   The regression shares one cache across two dictionary indexes with different 
ID mappings and verifies a single parse with correct segment-specific matches. 
All 591 focused tests and required affected-module checks passed under JDK 25.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to