Jackie-Jiang commented on code in PR #18758:
URL: https://github.com/apache/pinot/pull/18758#discussion_r4558702606


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/text/LuceneTextIndexReader.java:
##########
@@ -214,9 +220,24 @@ private File getTextIndexFile(File segmentIndexDir) {
     return file;
   }
 
+  @Override
+  public boolean isBuildOnDictionary() {
+    return _buildOnDictionary;
+  }
+
   @Override
   public ImmutableRoaringBitmap getDictIds(String searchQuery) {
-    throw new UnsupportedOperationException("");
+    return getDictIds(searchQuery, null);
+  }
+
+  @Override
+  public ImmutableRoaringBitmap getDictIds(String searchQuery, @Nullable 
String optionsString) {
+    if (_buildOnDictionary) {
+      // The translator resolves Lucene document IDs to the original 
dictionary IDs, including after merges.
+      return getDocIds(searchQuery, optionsString, null);

Review Comment:
   Optional improvement: preserve the existing query-scoped parser cache on the 
dictionary path. Passing `null` here means identical queries are parsed again 
for each segment, whereas the ordinary TEXT filter passes its query-scoped 
cache. A backward-compatible `getDictIds(query, options, cache)` overload could 
forward the evaluator factory's existing `QueryContext` and reuse the current 
parsing-cache implementation.



##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/TextMatchTransformFunction.java:
##########
@@ -106,19 +112,46 @@ public void init(List<TransformFunction> arguments, 
Map<String, ColumnContext> c
     }
 
     _textIndexReader = indexReader;
+    _columnArg = columnArg;
+    _dictionary = dataSource.getDictionary();
+    _matchingRawValues = null;
   }
 
   public int[] transformToIntValuesSV(ValueBlock valueBlock) {
     int length = valueBlock.getNumDocs();
     initZeroFillingIntValuesSV(length);
 
-    int[] docIds = valueBlock.getDocIds();
-    MutableRoaringBitmap indexDocIds = _textIndexReader.isMultiColumn()
-        ? ((MultiColumnTextIndexReader) _textIndexReader).getDocIds(_column, 
_predicate, _options)
-        : _textIndexReader.getDocIds(_predicate, _options);
+    int[] ids;
+    ImmutableRoaringBitmap matchingIds;
+    if (_textIndexReader.isBuildOnDictionary()) {
+      if (_columnArg.getDictionary() == null) {
+        // A standalone dictionary can coexist with a raw forward index.
+        if (_matchingRawValues == null) {
+          matchingIds = _textIndexReader.getDictIds(_predicate, _options);
+          _matchingRawValues = new 
ObjectOpenHashSet<>(matchingIds.getCardinality());
+          for (int dictId : matchingIds.toArray()) {
+            _matchingRawValues.add(_dictionary.getStringValue(dictId));
+          }
+        }
+        String[] values = _columnArg.transformToStringValuesSV(valueBlock);
+        for (int i = 0; i < length; i++) {
+          if (_matchingRawValues.contains(values[i])) {
+            _intValuesSV[i] = 1;
+          }
+        }
+        return _intValuesSV;
+      }
+      matchingIds = _textIndexReader.getDictIds(_predicate, _options);

Review Comment:
   Suggestion: lazily cache the matching dictionary-ID bitmap per transform 
instance and reset it in `init()`. This currently runs a Lucene search and 
allocates a bitmap/collector for every value block, although dictionary mode is 
immutable and the predicate/options are fixed. The raw-forward branch already 
caches its matching strings. This would make a large projection or 
`SUM(TEXT_MATCH(...))` search once per segment rather than once per batch. 
Please include a projection test exceeding 10,000 rows; the current 1,000-row 
tests do not cross a batch boundary.



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/TextIndexHandler.java:
##########
@@ -336,7 +336,15 @@ private void 
createTextIndexForColumn(SegmentDirectory.Writer segmentWriter, Col
         _fieldIndexConfigs.get(columnMetadata.getColumnName()), 
columnMetadata);
         ForwardIndexReaderContext readerContext = 
forwardIndexReader.createContext();
         TextIndexCreator textIndexCreator = 
StandardIndexes.text().createIndexCreator(context, config)) {
-      if (columnMetadata.isSingleValue()) {
+      if (config.isBuildOnDictionary()) {

Review Comment:
   Suggestion: branch on dictionary mode before `createForwardIndexIfNeeded()` 
and before opening the forward reader/context. This branch only reads 
dictionary values, so it does not need a forward index. For a forward-disabled 
column with an inverted index, the current setup reconstructs, serializes, and 
later deletes an entire temporary forward index unnecessarily. Dictionary mode 
can own just the dictionary and text creator, while marker handling, sealing, 
and packing remain shared. A forward-disabled reload case would cover this.



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/text/LuceneTextIndexCreator.java:
##########
@@ -153,7 +160,10 @@ public LuceneTextIndexCreator(String column, File 
segmentIndexDir, boolean commi
       // 1. Not the realtime index, i.e. commit is set to true
       // 2. Happens during realtime segment conversion
       // 3. Mutable segment not compacted
-      _reuseMutableIndex = config.isReuseMutableIndex() && commit && 
realtimeConversion && !mutableSegmentCompacted;
+      // 4. Not a dictionary-based index: the mutable index is per-row, so a 
dictionary-based immutable index must be
+      //    rebuilt from the sealed dictionary rather than reusing the mutable 
index.
+      _reuseMutableIndex = config.isReuseMutableIndex() && commit && 
realtimeConversion && !mutableSegmentCompacted
+          && !_buildOnDictionary;

Review Comment:
   [P1] Preserve original text before disabling mutable-index reuse
   
   With `buildOnDictionary=true`, `reuseMutableIndex=true`, and 
`indexes.text.rawValue` configured (or legacy `noRawDataForTextIndex=true`), 
`MutableSegmentImpl` still installs `SameValueMutableDictionary`, which retains 
only the placeholder. Disabling reuse here makes `TextIndexType` rebuild the 
text index from that placeholder, losing the original searchable terms. For 
input `alpha`/`beta` and replacement `n`, `TEXT_MATCH(column, 'alpha')` matches 
while consuming but stops matching after sealing. The new conversion test 
enables reuse but omits replacement values.
   
   Please reject this incompatible configuration or preserve original 
dictionary values during ingestion, and add the combination to the conversion 
regression.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to