xiangfu0 commented on code in PR #18758:
URL: https://github.com/apache/pinot/pull/18758#discussion_r4594453971


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/TextIndexHandler.java:
##########
@@ -331,17 +331,28 @@ private void 
createTextIndexForColumn(SegmentDirectory.Writer segmentWriter, Col
         .withTextCommitOnClose(true)
         .build();
 
-    IndexReaderFactory<ForwardIndexReader> readerFactory = 
StandardIndexes.forward().getReaderFactory();
-    try (ForwardIndexReader forwardIndexReader = 
readerFactory.createIndexReader(segmentWriter,
-        _fieldIndexConfigs.get(columnMetadata.getColumnName()), 
columnMetadata);
-        ForwardIndexReaderContext readerContext = 
forwardIndexReader.createContext();
-        TextIndexCreator textIndexCreator = 
StandardIndexes.text().createIndexCreator(context, config)) {
-      if (columnMetadata.isSingleValue()) {
-        processSVField(segmentWriter, hasDictionary, forwardIndexReader, 
readerContext, textIndexCreator, numDocs,
-            columnMetadata);
+    try (TextIndexCreator textIndexCreator = 
StandardIndexes.text().createIndexCreator(context, config)) {
+      if (config.isBuildOnDictionary()) {
+        // Dictionary-based text index: build one Lucene document per distinct 
value, storing its dictId,
+        // matching the dictionary order used by FSTIndexHandler. Requires a 
dictionary (validated upstream).
+        try (Dictionary dictionary = DictionaryIndexType.read(segmentWriter, 
columnMetadata)) {
+          for (int dictId = 0; dictId < dictionary.length(); dictId++) {
+            textIndexCreator.add(dictionary.getStringValue(dictId));

Review Comment:
   Fixed in `13bd1e68dd`. Reload checks the source before removing the old 
per-row index. Old realtime segments don't record whether they replaced the 
stored text, so a singleton source with a per-row index requires refresh or 
reingestion from the original data.
   
   The native and legacy regression cases seal under the old configuration, 
remove the placeholder setting, and verify that reload refuses the conversion 
and the original `alpha` matches are still queryable. The legacy case also 
covers a partial Lucene directory alongside the original packed index. Both 
cases failed on the prior code and passed in the 233-case 
`RealtimeSegmentConverterTest` run on JDK 25.



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/TextIndexHandler.java:
##########
@@ -308,11 +308,11 @@ private void 
createTextIndexForColumn(SegmentDirectory.Writer segmentWriter, Col
     int numDocs = columnMetadata.getTotalDocs();
     boolean hasDictionary = columnMetadata.hasDictionary();
 
-    // Create a temporary forward index if it is disabled and does not exist
-    columnMetadata = createForwardIndexIfNeeded(segmentWriter, columnName, 
true);
-
-    // Get text index configuration to check if storeInSegmentFile is enabled
     TextIndexConfig config = 
_fieldIndexConfigs.get(columnName).getConfig(StandardIndexes.text());
+    if (!config.isBuildOnDictionary()) {
+      // Row-based text indexes need a temporary forward index when it is 
disabled.
+      columnMetadata = createForwardIndexIfNeeded(segmentWriter, columnName, 
true);

Review Comment:
   Fixed in `13bd1e68dd`. Recovery now requires the dictionary for dictionary 
mode and the original forward index for per-row mode.
   
   `testDictionaryTextReloadWithoutForwardIndex` leaves a build marker on a 
segment without a forward index, then verifies that startup reload rebuilds the 
dictionary text index, returns the expected matches, and removes the marker. 
The prior code failed on the missing forward index. The regression and the 19 
rollback/recovery cases passed on JDK 25.



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/text/TextIndexType.java:
##########
@@ -103,8 +111,34 @@ protected ColumnConfigDeserializer<TextIndexConfig> 
createDeserializerForLegacyC
   @Override
   public TextIndexCreator createIndexCreator(IndexCreationContext context, 
TextIndexConfig indexConfig)
       throws IOException {
-    
Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() 
== FieldSpec.DataType.STRING,
+    
Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() 
== DataType.STRING,
         "Text index is currently only supported on STRING type columns");
+    if (indexConfig.isBuildOnDictionary()) {
+      Preconditions.checkState(context.getFieldSpec().isSingleValueField(),
+          "buildOnDictionary TEXT index requires a single-value column: %s", 
context.getFieldSpec().getName());
+      Preconditions.checkState(context.hasDictionary(),
+          "buildOnDictionary TEXT index requires a dictionary-encoded column: 
%s", context.getFieldSpec().getName());
+      // Columnar segment-creation / realtime-conversion path: build one 
Lucene document per distinct dictionary value
+      // (the stored id is the dictId). On the reload path ColumnStatistics is 
null and TextIndexHandler feeds the
+      // dictionary values via add(String) instead.
+      ColumnStatistics columnStatistics = context.getColumnStatistics();
+      if (columnStatistics != null) {
+        Object uniqueValues = columnStatistics.getUniqueValuesSet();
+        Preconditions.checkState(uniqueValues instanceof String[],
+            "Expected String[] dictionary values for dictionary-based TEXT 
index on column: %s",
+            context.getFieldSpec().getName());
+        String[] dictionaryValues = (String[]) uniqueValues;
+        Preconditions.checkState(dictionaryValues.length == 
context.getCardinality(),
+            "Dictionary cardinality mismatch for TEXT index on column: %s: 
expected %s values, got %s",
+            context.getFieldSpec().getName(), context.getCardinality(), 
dictionaryValues.length);
+        // BaseSegmentCreator builds the dictionary from this same sorted 
array, so array offsets are dictionary ids.
+        LuceneTextIndexCreator creator = new LuceneTextIndexCreator(context, 
indexConfig);
+        for (String value : dictionaryValues) {
+          creator.add(value);

Review Comment:
   Fixed in `13bd1e68dd`. If dictionary population throws, the factory rolls 
back the Lucene writer and closes its directory without publishing the index. 
Any cleanup failure is attached to the original exception.
   
   The regression reproduced the retained Lucene write lock on the prior code 
and verifies the lock is released, the original exception is preserved, and no 
commit or combined index is published. All 25 `LuceneTextIndexCreatorTest` 
cases passed on JDK 25.



##########
pinot-core/src/test/java/org/apache/pinot/queries/DictionaryBasedTextIndexQueriesTest.java:
##########
@@ -0,0 +1,570 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.queries;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.exception.ExceptionUtils;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.core.common.Operator;
+import org.apache.pinot.core.operator.blocks.results.AggregationResultsBlock;
+import org.apache.pinot.core.operator.blocks.results.SelectionResultsBlock;
+import 
org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader;
+import 
org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.readers.GenericRowRecordReader;
+import org.apache.pinot.segment.local.utils.TableConfigUtils;
+import org.apache.pinot.segment.spi.ImmutableSegment;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.config.table.FieldConfig.EncodingType;
+import org.apache.pinot.spi.config.table.FieldConfig.IndexType;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.data.readers.RecordReader;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.DataProvider;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertNull;
+import static org.testng.Assert.assertTrue;
+import static org.testng.Assert.fail;
+
+
+/// Verifies that a dictionary-based TEXT index (`buildOnDictionary = true`) 
returns the same `TEXT_MATCH`
+/// results as the default per-row TEXT index for single-value STRING columns, 
while multi-value columns retain
+/// the per-row index. The dictionary-based index returns matching dictIds 
resolved to docIds by dictionary-based
+/// filter operators; this test confirms that resolution is correct end-to-end.
+public class DictionaryBasedTextIndexQueriesTest extends BaseQueriesTest {
+  private static final File INDEX_DIR =
+      new File(FileUtils.getTempDirectory(), 
DictionaryBasedTextIndexQueriesTest.class.getSimpleName());
+  private static final String TABLE_NAME = "testTable";
+  private static final String SV_COL = "SV_TEXT";
+  private static final String MV_COL = "MV_TEXT";
+  private static final String INT_COL = "INT_COL";
+  private static final int NUM_ROWS = 1000;
+
+  // SV value cycles over 5 distinct tokens; MV value pairs a constant token 
with a cycling token.
+  private static final String[] SV_TOKENS = {"apple", "banana", "cherry", 
"date", "elderberry"};
+  private static final String[] MV_TOKENS = {"red", "green", "blue"};
+
+  private static final Schema SCHEMA = new 
Schema.SchemaBuilder().setSchemaName(TABLE_NAME)
+      .addSingleValueDimension(SV_COL, FieldSpec.DataType.STRING)
+      .addMultiValueDimension(MV_COL, FieldSpec.DataType.STRING)
+      .addMetric(INT_COL, FieldSpec.DataType.INT)
+      .build();
+
+  private IndexSegment _indexSegment;
+  private List<IndexSegment> _indexSegments;
+
+  @Override
+  protected String getFilter() {
+    return "";
+  }
+
+  @Override
+  protected IndexSegment getIndexSegment() {
+    return _indexSegment;
+  }
+
+  @Override
+  protected List<IndexSegment> getIndexSegments() {
+    return _indexSegments;
+  }
+
+  @BeforeClass
+  public void setUp() {
+    FileUtils.deleteQuietly(INDEX_DIR);
+  }
+
+  @AfterClass
+  public void tearDown() {
+    if (_indexSegment != null) {
+      _indexSegment.destroy();
+    }
+    FileUtils.deleteQuietly(INDEX_DIR);
+  }
+
+  private static List<GenericRow> createTestData() {
+    return createTestData(NUM_ROWS);
+  }
+
+  private static List<GenericRow> createTestData(int numRows) {
+    List<GenericRow> rows = new ArrayList<>(numRows);
+    for (int i = 0; i < numRows; i++) {
+      GenericRow row = new GenericRow();
+      row.putValue(SV_COL, SV_TOKENS[i % SV_TOKENS.length]);
+      row.putValue(MV_COL, new String[]{"fruit", MV_TOKENS[i % 
MV_TOKENS.length]});
+      row.putValue(INT_COL, i);
+      rows.add(row);
+    }
+    return rows;
+  }
+
+  private TableConfig tableConfig(boolean buildOnDictionary) {
+    return tableConfig(buildOnDictionary, false);
+  }
+
+  private TableConfig tableConfig(boolean buildOnDictionary, boolean 
storeInSegmentFile) {
+    Map<String, String> props = Map.of(
+        FieldConfig.TEXT_INDEX_BUILD_ON_DICTIONARY, 
Boolean.toString(buildOnDictionary),
+        "storeInSegmentFile", Boolean.toString(storeInSegmentFile));
+    List<FieldConfig> fieldConfigs = List.of(
+        new FieldConfig(SV_COL, EncodingType.DICTIONARY, 
List.of(IndexType.TEXT), null, props),

Review Comment:
   Updated in `13bd1e68dd`. The query test helper now configures 
`buildOnDictionary` and `storeInSegmentFile` through `indexes.text`. The 
raw-forward variant retains those settings and asserts that it still uses 
dictionary mode. The realtime converter tests retain coverage of the legacy 
properties.
   
   All 13 `DictionaryBasedTextIndexQueriesTest` cases and all 233 
`RealtimeSegmentConverterTest` cases passed on JDK 25.



##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/TextMatchTransformFunction.java:
##########
@@ -106,19 +113,49 @@ public void init(List<TransformFunction> arguments, 
Map<String, ColumnContext> c
     }
 
     _textIndexReader = indexReader;
+    _columnArg = columnArg;
+    _dictionary = dataSource.getDictionary();
+    _matchingRawValues = null;
+    _matchingDictIds = null;
   }
 
   public int[] transformToIntValuesSV(ValueBlock valueBlock) {
     int length = valueBlock.getNumDocs();
     initZeroFillingIntValuesSV(length);
 
-    int[] docIds = valueBlock.getDocIds();
-    MutableRoaringBitmap indexDocIds = _textIndexReader.isMultiColumn()
-        ? ((MultiColumnTextIndexReader) _textIndexReader).getDocIds(_column, 
_predicate, _options)
-        : _textIndexReader.getDocIds(_predicate, _options);
+    int[] ids;
+    ImmutableRoaringBitmap matchingIds;
+    if (_textIndexReader.isBuildOnDictionary()) {
+      if (_matchingDictIds == null) {
+        _matchingDictIds = _textIndexReader.getDictIds(_predicate, _options);
+      }
+      matchingIds = _matchingDictIds;
+      if (_columnArg.getDictionary() == null) {
+        // A standalone dictionary can coexist with a raw forward index.
+        if (_matchingRawValues == null) {
+          _matchingRawValues = new 
ObjectOpenHashSet<>(matchingIds.getCardinality());
+          for (int dictId : matchingIds.toArray()) {

Review Comment:
   Updated in `13bd1e68dd`. The raw-forward projection now iterates matching 
dictionary IDs directly with the primitive bitmap iterator, so it avoids 
allocating a temporary array.
   
   All 13 `DictionaryBasedTextIndexQueriesTest` cases, including the 
raw-forward projection, and all 10 `TextMatchTransformFunctionTest` cases 
passed on JDK 25.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to