Jackie-Jiang commented on code in PR #18758:
URL: https://github.com/apache/pinot/pull/18758#discussion_r4593942689


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/TextIndexHandler.java:
##########
@@ -331,17 +331,28 @@ private void 
createTextIndexForColumn(SegmentDirectory.Writer segmentWriter, Col
         .withTextCommitOnClose(true)
         .build();
 
-    IndexReaderFactory<ForwardIndexReader> readerFactory = 
StandardIndexes.forward().getReaderFactory();
-    try (ForwardIndexReader forwardIndexReader = 
readerFactory.createIndexReader(segmentWriter,
-        _fieldIndexConfigs.get(columnMetadata.getColumnName()), 
columnMetadata);
-        ForwardIndexReaderContext readerContext = 
forwardIndexReader.createContext();
-        TextIndexCreator textIndexCreator = 
StandardIndexes.text().createIndexCreator(context, config)) {
-      if (columnMetadata.isSingleValue()) {
-        processSVField(segmentWriter, hasDictionary, forwardIndexReader, 
readerContext, textIndexCreator, numDocs,
-            columnMetadata);
+    try (TextIndexCreator textIndexCreator = 
StandardIndexes.text().createIndexCreator(context, config)) {
+      if (config.isBuildOnDictionary()) {
+        // Dictionary-based text index: build one Lucene document per distinct 
value, storing its dictId,
+        // matching the dictionary order used by FSTIndexHandler. Requires a 
dictionary (validated upstream).
+        try (Dictionary dictionary = DictionaryIndexType.read(segmentWriter, 
columnMetadata)) {
+          for (int dictId = 0; dictId < dictionary.length(); dictId++) {
+            textIndexCreator.add(dictionary.getStringValue(dictId));

Review Comment:
   [P1] Protect existing placeholder dictionaries during migration
   
   Existing realtime segments sealed with `buildOnDictionary=false`, 
`reuseMutableIndex=true`, and `indexes.text.rawValue` (or legacy 
`noRawDataForTextIndex`) can contain only placeholders in their dictionaries 
while retaining the original terms in their per-row Lucene indexes. Enabling 
`buildOnDictionary` and reloading deletes that valid text index, then rebuilds 
from placeholders here, silently losing matches. The ingestion fix preserves 
values for newly created consuming segments but cannot restore previously 
discarded values.
   
   Please validate the source before replacing the existing index, reject 
conversion and require refresh/reingestion when the original text is 
unavailable, and add a regression that seals under the old configuration before 
enabling dictionary mode.



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/TextIndexHandler.java:
##########
@@ -308,11 +308,11 @@ private void 
createTextIndexForColumn(SegmentDirectory.Writer segmentWriter, Col
     int numDocs = columnMetadata.getTotalDocs();
     boolean hasDictionary = columnMetadata.hasDictionary();
 
-    // Create a temporary forward index if it is disabled and does not exist
-    columnMetadata = createForwardIndexIfNeeded(segmentWriter, columnName, 
true);
-
-    // Get text index configuration to check if storeInSegmentFile is enabled
     TextIndexConfig config = 
_fieldIndexConfigs.get(columnName).getConfig(StandardIndexes.text());
+    if (!config.isBuildOnDictionary()) {
+      // Row-based text indexes need a temporary forward index when it is 
disabled.
+      columnMetadata = createForwardIndexIfNeeded(segmentWriter, columnName, 
true);

Review Comment:
   [P2] Allow dictionary-only builds to recover without a forward index
   
   Dictionary mode now skips forward-index creation, but 
`validateRecoverySource()` still requires a forward index unconditionally. If 
startup preprocessing stops after writing `.text.inprogress`, the next load 
rejects recovery despite an intact dictionary, potentially forcing a segment 
download. This affects preprocessing in place on startup, where the explicit 
reload backup does not protect the build.
   
   Please validate the required source according to the target build mode 
(dictionary for dictionary mode, original forward index for per-row mode), and 
add an interrupted-build case to `testDictionaryTextReloadWithoutForwardIndex`.



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/text/TextIndexType.java:
##########
@@ -103,8 +111,34 @@ protected ColumnConfigDeserializer<TextIndexConfig> 
createDeserializerForLegacyC
   @Override
   public TextIndexCreator createIndexCreator(IndexCreationContext context, 
TextIndexConfig indexConfig)
       throws IOException {
-    
Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() 
== FieldSpec.DataType.STRING,
+    
Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() 
== DataType.STRING,
         "Text index is currently only supported on STRING type columns");
+    if (indexConfig.isBuildOnDictionary()) {
+      Preconditions.checkState(context.getFieldSpec().isSingleValueField(),
+          "buildOnDictionary TEXT index requires a single-value column: %s", 
context.getFieldSpec().getName());
+      Preconditions.checkState(context.hasDictionary(),
+          "buildOnDictionary TEXT index requires a dictionary-encoded column: 
%s", context.getFieldSpec().getName());
+      // Columnar segment-creation / realtime-conversion path: build one 
Lucene document per distinct dictionary value
+      // (the stored id is the dictId). On the reload path ColumnStatistics is 
null and TextIndexHandler feeds the
+      // dictionary values via add(String) instead.
+      ColumnStatistics columnStatistics = context.getColumnStatistics();
+      if (columnStatistics != null) {
+        Object uniqueValues = columnStatistics.getUniqueValuesSet();
+        Preconditions.checkState(uniqueValues instanceof String[],
+            "Expected String[] dictionary values for dictionary-based TEXT 
index on column: %s",
+            context.getFieldSpec().getName());
+        String[] dictionaryValues = (String[]) uniqueValues;
+        Preconditions.checkState(dictionaryValues.length == 
context.getCardinality(),
+            "Dictionary cardinality mismatch for TEXT index on column: %s: 
expected %s values, got %s",
+            context.getFieldSpec().getName(), context.getCardinality(), 
dictionaryValues.length);
+        // BaseSegmentCreator builds the dictionary from this same sorted 
array, so array offsets are dictionary ids.
+        LuceneTextIndexCreator creator = new LuceneTextIndexCreator(context, 
indexConfig);
+        for (String value : dictionaryValues) {
+          creator.add(value);

Review Comment:
   Suggestion: make dictionary population exception-safe. If `creator.add()` 
throws (for example, an analyzer or I/O failure), this factory has not 
returned, so the caller has not registered the creator and its cleanup cannot 
release the Lucene writer/directory. Add failure cleanup here while preserving 
the original exception. The failure path should release resources without 
treating a partially populated index as successfully built.



##########
pinot-core/src/test/java/org/apache/pinot/queries/DictionaryBasedTextIndexQueriesTest.java:
##########
@@ -0,0 +1,570 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.queries;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.exception.ExceptionUtils;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.core.common.Operator;
+import org.apache.pinot.core.operator.blocks.results.AggregationResultsBlock;
+import org.apache.pinot.core.operator.blocks.results.SelectionResultsBlock;
+import 
org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader;
+import 
org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.readers.GenericRowRecordReader;
+import org.apache.pinot.segment.local.utils.TableConfigUtils;
+import org.apache.pinot.segment.spi.ImmutableSegment;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.config.table.FieldConfig.EncodingType;
+import org.apache.pinot.spi.config.table.FieldConfig.IndexType;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.data.readers.RecordReader;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.DataProvider;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertNull;
+import static org.testng.Assert.assertTrue;
+import static org.testng.Assert.fail;
+
+
+/// Verifies that a dictionary-based TEXT index (`buildOnDictionary = true`) 
returns the same `TEXT_MATCH`
+/// results as the default per-row TEXT index for single-value STRING columns, 
while multi-value columns retain
+/// the per-row index. The dictionary-based index returns matching dictIds 
resolved to docIds by dictionary-based
+/// filter operators; this test confirms that resolution is correct end-to-end.
+public class DictionaryBasedTextIndexQueriesTest extends BaseQueriesTest {
+  private static final File INDEX_DIR =
+      new File(FileUtils.getTempDirectory(), 
DictionaryBasedTextIndexQueriesTest.class.getSimpleName());
+  private static final String TABLE_NAME = "testTable";
+  private static final String SV_COL = "SV_TEXT";
+  private static final String MV_COL = "MV_TEXT";
+  private static final String INT_COL = "INT_COL";
+  private static final int NUM_ROWS = 1000;
+
+  // SV value cycles over 5 distinct tokens; MV value pairs a constant token 
with a cycling token.
+  private static final String[] SV_TOKENS = {"apple", "banana", "cherry", 
"date", "elderberry"};
+  private static final String[] MV_TOKENS = {"red", "green", "blue"};
+
+  private static final Schema SCHEMA = new 
Schema.SchemaBuilder().setSchemaName(TABLE_NAME)
+      .addSingleValueDimension(SV_COL, FieldSpec.DataType.STRING)
+      .addMultiValueDimension(MV_COL, FieldSpec.DataType.STRING)
+      .addMetric(INT_COL, FieldSpec.DataType.INT)
+      .build();
+
+  private IndexSegment _indexSegment;
+  private List<IndexSegment> _indexSegments;
+
+  @Override
+  protected String getFilter() {
+    return "";
+  }
+
+  @Override
+  protected IndexSegment getIndexSegment() {
+    return _indexSegment;
+  }
+
+  @Override
+  protected List<IndexSegment> getIndexSegments() {
+    return _indexSegments;
+  }
+
+  @BeforeClass
+  public void setUp() {
+    FileUtils.deleteQuietly(INDEX_DIR);
+  }
+
+  @AfterClass
+  public void tearDown() {
+    if (_indexSegment != null) {
+      _indexSegment.destroy();
+    }
+    FileUtils.deleteQuietly(INDEX_DIR);
+  }
+
+  private static List<GenericRow> createTestData() {
+    return createTestData(NUM_ROWS);
+  }
+
+  private static List<GenericRow> createTestData(int numRows) {
+    List<GenericRow> rows = new ArrayList<>(numRows);
+    for (int i = 0; i < numRows; i++) {
+      GenericRow row = new GenericRow();
+      row.putValue(SV_COL, SV_TOKENS[i % SV_TOKENS.length]);
+      row.putValue(MV_COL, new String[]{"fruit", MV_TOKENS[i % 
MV_TOKENS.length]});
+      row.putValue(INT_COL, i);
+      rows.add(row);
+    }
+    return rows;
+  }
+
+  private TableConfig tableConfig(boolean buildOnDictionary) {
+    return tableConfig(buildOnDictionary, false);
+  }
+
+  private TableConfig tableConfig(boolean buildOnDictionary, boolean 
storeInSegmentFile) {
+    Map<String, String> props = Map.of(
+        FieldConfig.TEXT_INDEX_BUILD_ON_DICTIONARY, 
Boolean.toString(buildOnDictionary),
+        "storeInSegmentFile", Boolean.toString(storeInSegmentFile));
+    List<FieldConfig> fieldConfigs = List.of(
+        new FieldConfig(SV_COL, EncodingType.DICTIONARY, 
List.of(IndexType.TEXT), null, props),

Review Comment:
   Optional cleanup: use the current `indexes.text` JSON configuration in the 
main test helper, with `buildOnDictionary` and `storeInSegmentFile`, matching 
the documented usage. These feature tests currently go through legacy 
`indexTypes`/`properties`; using the current configuration by default and 
keeping a focused legacy compatibility case would exercise the intended 
configuration path more directly.



##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/TextMatchTransformFunction.java:
##########
@@ -106,19 +113,49 @@ public void init(List<TransformFunction> arguments, 
Map<String, ColumnContext> c
     }
 
     _textIndexReader = indexReader;
+    _columnArg = columnArg;
+    _dictionary = dataSource.getDictionary();
+    _matchingRawValues = null;
+    _matchingDictIds = null;
   }
 
   public int[] transformToIntValuesSV(ValueBlock valueBlock) {
     int length = valueBlock.getNumDocs();
     initZeroFillingIntValuesSV(length);
 
-    int[] docIds = valueBlock.getDocIds();
-    MutableRoaringBitmap indexDocIds = _textIndexReader.isMultiColumn()
-        ? ((MultiColumnTextIndexReader) _textIndexReader).getDocIds(_column, 
_predicate, _options)
-        : _textIndexReader.getDocIds(_predicate, _options);
+    int[] ids;
+    ImmutableRoaringBitmap matchingIds;
+    if (_textIndexReader.isBuildOnDictionary()) {
+      if (_matchingDictIds == null) {
+        _matchingDictIds = _textIndexReader.getDictIds(_predicate, _options);
+      }
+      matchingIds = _matchingDictIds;
+      if (_columnArg.getDictionary() == null) {
+        // A standalone dictionary can coexist with a raw forward index.
+        if (_matchingRawValues == null) {
+          _matchingRawValues = new 
ObjectOpenHashSet<>(matchingIds.getCardinality());
+          for (int dictId : matchingIds.toArray()) {

Review Comment:
   Small cleanup: iterate `matchingIds` with its primitive bitmap iterator 
instead of `toArray()`. The array is discarded immediately after populating the 
matching-string set, so direct iteration avoids an allocation proportional to 
the number of matching dictionary values.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to