xiangfu0 opened a new pull request, #18758: URL: https://github.com/apache/pinot/pull/18758
## Description Adds an opt-in **`buildOnDictionary`** flag (default **false**) for the Lucene **TEXT** index. When enabled on a dictionary-encoded `STRING` column, the Lucene index is built over the column **dictionary** — one document per distinct value, with the Lucene docId equal to the dictId — instead of one document per row. This makes the text index size scale with **cardinality** rather than **row count**, which is a large win for low/medium-cardinality text columns, while keeping the full Lucene query surface. `TEXT_MATCH` returns the matching dictIds, which are resolved to docIds through the existing dictionary-based filter operators (inverted index / scan) — mirroring the FST / `REGEXP_LIKE` path. Existing behavior is unchanged when the flag is off. ### Design (mirrors the FST dictionary pattern) - **Config:** `FieldConfig.TEXT_INDEX_BUILD_ON_DICTIONARY` + `TextIndexConfig.isBuildOnDictionary()` (default false), parsed in `TextIndexConfigBuilder`. - **Creation:** `TextIndexType.createIndexCreator` feeds the sorted dictionary values to a dictionary-mode `LuceneTextIndexCreator` (the per-row `add()` becomes a no-op). Dictionary mode is gated to `commit == true`, so **consuming (realtime) segments keep building per-row** and the dictionary-based index is produced on seal/convert and on offline ingestion. Reload is handled by `TextIndexHandler` looping the dictionary, mirroring `FSTIndexHandler`. - **Self-describing segments (mixed-table safe):** the build mode is persisted into the segment's `lucene.properties`, and the reader derives its mode from the **segment**, not the live table config. A table holding a **mix** of dictionary-based and per-row text segments (e.g. during rollout) reads each correctly. The table-config flag governs **creation only**. - **Reader:** `LuceneTextIndexReader` serves `getDictIds()` in dictionary mode and uses a no-op docId translator (Lucene docId already equals dictId). - **Query:** new `TextMatchDictIdPredicateEvaluatorFactory`; `FilterPlanNode` routes a dictionary-based `TEXT_MATCH` through `FilterOperatorUtils` to the standard scan/inverted operators. **Single-value and multi-value** STRING columns supported. - **Validation (hard failure):** `buildOnDictionary` on a non-dictionary-encoded column is rejected at table-config validation and again by a creation-time precondition. ### Limitations / follow-ups - `TEXT_MATCH` **options** are not supported in dictionary mode (the dictionary lookup takes only the query value) — such queries are rejected rather than silently ignored. - Realtime consuming segments always build the per-row index; the dictionary-based index is produced when the segment is sealed/converted. ## Upgrade Notes Opt-in and default-off, so no behavior change on upgrade. The build mode is recorded per segment, so a table can hold a mix of dictionary-based and per-row text segments during a rolling enablement and each is read correctly. Flipping the flag triggers a text-index rebuild on segment reload. ## Testing `DictionaryBasedTextIndexQueriesTest` (new): - dictionary-based `TEXT_MATCH` results equal the per-row index for SV and MV columns, including `OR` and zero-match queries; - reload builds the dictionary-based index from the dictionary; - the dictionary-based segment is smaller than per-row for a low-cardinality column; - a non-dictionary column with `buildOnDictionary=true` fails the build; - `TEXT_MATCH` with options is rejected for a dictionary-based index. `TextIndexConfigTest`: round-trips the new flag. Regression (all green): per-row `TextSearchQueriesTest`, `FSTBasedRegexpLikeQueriesTest`, `IFSTBasedRegexpLikeQueriesTest`, and segment-local `LuceneTextIndexCreatorTest` / `TextIndexTest` / `LuceneTextIndexConfigReloadTest` / `TextIndexUtilsTest`. `spotless` / `license` / `checkstyle` clean. ## Labels `feature`, `text-index` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
