danny0405 commented on code in PR #19613: URL: https://github.com/apache/hudi/pull/19613#discussion_r3789250437
########## rfc/rfc-full-text-search/rfc-full-text-search.md: ########## @@ -0,0 +1,891 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# RFC: Hudi Full-Text Search Index + +## Proposers + +- @danny0405 + +## Approvers + +- TBD + +## Status + +Issue: TBD + +> This proposal intentionally does not claim an RFC number. The number and +> catalog entry will be reserved through the separate RFC-number process. + +## Abstract + +This RFC proposes a full-text index for Apache Hudi. It supports token, phrase, +and multi-column search over string columns, exposes standard predicates through +Spark SQL, and offers a Lance-style direct table API through Hudi RS Python. +Indexes are built and maintained through the Hudi metadata table (MDT) +indexing lifecycle. + +The search engine follows Lance's useful architectural choices without making +Lance a storage dependency: immutable segments, a compact term dictionary, +compressed posting lists, document-length statistics, BM25 ranking, positions, +and block-max WAND. Hudi owns the analyzer contract and on-disk format. Spark +integration remains in the main Hudi repository; Rust and Python implementation +work belongs in Hudi RS and consumes the same versioned format. + +The MDT remains authoritative for index definitions, visibility, coverage, +rollbacks, and cleaning. Large immutable posting payloads are sidecar files in +an auxiliary directory owned by the MDT rather than values embedded in HFiles. +An MDT commit atomically publishes descriptors for already durable payloads. + +Queries are snapshot-safe. SQL predicates always combine index results with a +raw scan of source file slices not covered by a compatible segment, so using an +index never changes SQL results. The direct search API also defaults to complete +results and may offer an explicitly incomplete low-latency mode. + +## Background + +Hudi indexes currently answer questions such as which files might contain a +record key or a value range. Full-text search has a different contract: analyze +free text into terms, locate matching documents, optionally verify positions, +and, for direct search APIs, rank the best documents. Sending this workload to +Elasticsearch or OpenSearch is effective, but creates a second ingestion +pipeline and a second source of snapshot and retention truth. + +This proposal builds on: + +- [RFC-45](../rfc-45/rfc-45.md), which introduced asynchronous MDT indexing; +- [RFC-77](../rfc-77/rfc-77.md), which established dynamically named secondary + index partitions and index definitions; +- the standard Spark SQL predicate model, which keeps index acceleration + transparent to relational queries; and +- RFC-109, the native vector-index proposal listed in the RFC catalog. Text and + vector search may share Hudi RS storage adapters and top-k utilities, but + their persistent formats remain independent. + +### Design principles + +1. The Hudi timeline is the source of snapshot truth. +2. Index creation, visibility, rollback, and cleaning use MDT components. +3. Immutable payloads support object-store range reads and safe caching. +4. Index use never changes the result of a Spark SQL predicate. +5. Ranking is independent of how the index is physically partitioned. +6. JVM and Hudi RS clients share query semantics and format versions. + +### Goals + +- Match token, boolean, prefix, fuzzy, phrase, and multi-column queries. +- Support copy-on-write (COW) and merge-on-read (MOR) tables. +- Build asynchronously and incrementally using the MDT indexer lifecycle. +- Guarantee snapshot-correct results for SQL and the default direct API mode. +- Provide an object-store-friendly text-index format with bounded memory usage. +- Provide a direct Hudi RS Python API alongside Spark SQL. + +### Non-goals + +- Elasticsearch API, aggregation, highlighting, or percolator compatibility. +- Highlighting and custom relevance models in the first format version. +- Updating posting lists in place. +- Replacing SQL predicate indexes or the record index. +- Adding Rust code or native build integration to the main Hudi repository. + +### Alternatives considered + +**External Elasticsearch/OpenSearch.** This remains a valid integration, but it +requires change-data-capture coordination, separate retention, and explicit +mapping between external documents and a Hudi snapshot. + +**Embedding Tantivy.** Tantivy is mature and Rust-native. Its archive and +directory abstractions, however, become a second persistent compatibility +contract. A smaller Hudi-owned format gives the project control over source +file-slice identity, range-read layout, and MDT publication semantics. + +**Posting lists as MDT record values.** This would make MDT storage atomic, but +multi-gigabyte postings, merges, and random term reads fight the metadata +table's record-oriented HFile/MOR strengths. Small authoritative descriptors in +the MDT plus immutable sidecars preserve the lifecycle benefits without that +cost. + +## Implementation + +### Terminology + +| Term | Meaning | +| --- | --- | +| Logical index | A named SQL index and its immutable analyzer configuration. | +| MDT index partition | Dynamic `text_index_<name>` metadata partition containing authoritative control records. | +| Segment | An immutable set of documents and term postings built together. | +| Payload partition | A shard within a segment whose local document identifiers are `u32`. | +| Source slice | A Hudi base file and its ordered log files at a snapshot. | +| Coverage | Proof that a segment represents a particular source-slice fingerprint. | +| Raw tail | Eligible source slices not covered by compatible visible segments. | +| Document | One Hudi record, identified by record key and source information. | + +### SQL interface + +Creation follows Hudi's secondary-index syntax: + +```sql +CREATE INDEX article_body_fts +ON articles +USING text_index (body) +OPTIONS ( + 'base_tokenizer' = 'simple', + 'lower_case' = 'true', + 'language' = 'und', + 'with_position' = 'true', + 'posting_block_size' = '128' +); +``` + +The existing `HoodieIndexDefinition` is populated as follows: + +```json +{ + "indexName": "article_body_fts", + "indexType": "text_index", + "sourceFields": ["body"], + "indexFunction": "tokenize", + "indexOptions": { + "base_tokenizer": "simple", + "lower_case": "true", + "language": "und", + "with_position": "true", + "posting_block_size": "128" + } +} +``` + +#### Spark SQL predicates + +The primary Spark interface follows the conventional index experience: a +boolean predicate in `WHERE`. Whether the optimizer uses the text index is not +observable in query results. + +```sql +SELECT _hoodie_record_key, title +FROM articles +WHERE hudi_match(body, 'lakehouse indexing', 'operator=AND'); +``` + +Phrase and multi-column queries use companion predicates: + +```sql +SELECT _hoodie_record_key, title +FROM articles +WHERE hudi_match_phrase(body, 'metadata table', 1) + AND category = 'engineering'; + +SELECT _hoodie_record_key, title +FROM articles +WHERE hudi_multi_match('lakehouse indexing', title, body); +``` + +The v1 signatures are: + +```text +hudi_match(column, query [, 'key=value,...']) -> boolean +hudi_match_phrase(column, query [, slop]) -> boolean +hudi_multi_match(query [, 'operator=AND|OR'], column, ...) -> boolean +``` + +`hudi_match` options include `operator`, `fuzziness`, `prefix_length`, and +`max_expansions`. Unknown options are rejected. The functions compose with +normal SQL predicates. Conjunctive text predicates can be pushed into index +planning; expressions whose `OR` semantics cannot be preserved are evaluated +by Spark without index pushdown. + +SQL predicate evaluation is always complete. Compatible segments accelerate +covered source slices, while uncovered or incompatible slices are evaluated by +the normal Hudi scan. There is no `_score` column and `LIMIT` does not imply +relevance order. Ranked top-k is a separate direct-search contract. + +#### Hudi RS Python table API + +Python users should not need to construct Spark SQL strings. Hudi RS extends +its existing `HudiTableBuilder` and `read_snapshot` API with the same structured +query model used by the index. The proposed predicate-style API is: + +```python +import pyarrow as pa + +from hudi import HudiTableBuilder +from hudi.search import FullTextOperator, MatchQuery, PhraseQuery + +table = ( + HudiTableBuilder + .from_base_uri("s3://warehouse/articles") + .build() +) + +query = MatchQuery( + "lakehouse indexing", + column="body", + operator=FullTextOperator.AND, +) + +batches = table.read_snapshot( + columns=["_hoodie_record_key", "title", "body"], + filters=[("category", "=", "engineering")], + full_text_query=query, +) +articles = pa.Table.from_batches(batches) +``` + +Structured queries compose without inventing a second query language: + +```python +query = ( + MatchQuery("metadata table", column="body") + & PhraseQuery("incremental indexing", column="body", slop=1) +) + +batches = table.read_snapshot(full_text_query=query) +``` + +For search applications, a Lance-style fluent API exposes ranked top-k and an +explicit score: + +```python +results = ( + table.search_text( + MatchQuery("lakehouse indexing", column="body"), + ) + .where([("category", "=", "engineering")]) + .select(["_hoodie_record_key", "title", "body"]) + .limit(20) + .to_arrow() +) + +# Ranked by BM25 descending; `_score` is included in `results`. +``` + +`read_snapshot(full_text_query=...)` has predicate semantics and returns every +match. `search_text(...).limit(k)` has ranked-search semantics and returns +`_score`. Both pin one Hudi snapshot, use identical analyzer/query objects, and +raw-scan uncovered source slices by default. The fluent API may expose +`allow_incomplete_index=True`, but it must mark the result metadata as +incomplete rather than silently changing defaults. + +Index creation remains an MDT table-service operation in the initial release, +invoked through Spark SQL or the Java API. A future Hudi RS writer API may add +`create_text_index` only after it can publish the corresponding MDT timeline +changes safely. + +The three query entry points share one query model and coverage planner, but +their result contracts differ: + +```mermaid +flowchart LR + SQL["Spark SQL<br/>hudi_match(...)"] + SNAPSHOT["Hudi RS Python<br/>read_snapshot(full_text_query=...)"] + SEARCH["Hudi RS Python<br/>search_text(...).limit(k)"] + MODEL["Shared query objects,<br/>analyzer contract, and snapshot pinning"] + MATCHES["Complete unordered<br/>match set"] + RANKED["BM25-ranked top-k<br/>with _score"] + + SQL --> MODEL + SNAPSHOT --> MODEL + SEARCH --> MODEL + MODEL -->|"predicate semantics"| MATCHES + MODEL -->|"ranked-search semantics"| RANKED +``` + +### Architecture + +```mermaid +flowchart TD + TIMELINE["Hudi data timeline<br/>snapshot S"] + DEFINITION["Index definition<br/>.hoodie/.index/index.json"] + MDT["MDT partition<br/>text_index_<name>"] + CONTROL["HEAD, SEGMENT, COVERAGE,<br/>and TOMBSTONE records"] + SIDECARS["Immutable sidecars<br/>tokens, docs, postings, positions"] + PLANNER["Snapshot coverage planner"] + INDEXED["Exactly covered<br/>source slices"] + RAW["Changed or uncovered<br/>raw source slices"] + INDEX_SCAN["Posting-list and<br/>position evaluation"] + RAW_SCAN["Normal Hudi scan with<br/>the same analyzer/query"] + UNION["Union and deduplicate<br/>document addresses"] + ROWS["Materialize rows and<br/>apply residual filters"] + RANK["Optional Hudi RS<br/>global BM25 top-k"] + + TIMELINE --> PLANNER + DEFINITION --> MDT + MDT --> CONTROL + CONTROL --> SIDECARS + CONTROL --> PLANNER + PLANNER --> INDEXED + PLANNER --> RAW + INDEXED --> INDEX_SCAN + SIDECARS --> INDEX_SCAN + RAW --> RAW_SCAN + INDEX_SCAN --> UNION + RAW_SCAN --> UNION + UNION --> ROWS + UNION --> RANK +``` + +The SQL definition is stored in `.hoodie/.index/index.json`. A dynamic MDT +partition stores small control records. Immutable native payload files live +under an MDT-owned auxiliary namespace. Readers never infer visibility by +listing that namespace; they use descriptors visible in the pinned MDT +snapshot. + +### Metadata table integration + +Add `TEXT_INDEX` to `MetadataPartitionType` with the dynamic prefix +`text_index_`. `getPartitionPath(metaClient, indexName)` and index-definition +lookup follow the secondary and expression index conventions. Add a +`TextSearchIndexer` to `IndexerFactory`, implementing `BaseIndexer` lifecycle +operations. + +Add a tagged Avro metadata payload named `HoodieTextIndexInfo` to +`HoodieMetadata.avsc`. The record is deliberately descriptor-sized and has +four logical kinds: + +| Kind | Record key | Purpose | +| --- | --- | --- | +| `HEAD` | `head` | Format, analyzer fingerprint, publication instant, aggregate stats. | +| `SEGMENT` | `segment/<uuid>` | Payload paths, sizes, checksums, statistics, source-mask cardinality. | Review Comment: Addressed. The RFC now specifies the many-file-group segment mapping through file_slice_ordinal and per-file-group COVERAGE records. It also explains how older and newer candidate segments coexist for time travel while the planner selects at most one exact match per query snapshot. ########## rfc/rfc-full-text-search/rfc-full-text-search.md: ########## @@ -0,0 +1,891 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# RFC: Hudi Full-Text Search Index + +## Proposers + +- @danny0405 + +## Approvers + +- TBD + +## Status + +Issue: TBD + +> This proposal intentionally does not claim an RFC number. The number and +> catalog entry will be reserved through the separate RFC-number process. + +## Abstract + +This RFC proposes a full-text index for Apache Hudi. It supports token, phrase, +and multi-column search over string columns, exposes standard predicates through +Spark SQL, and offers a Lance-style direct table API through Hudi RS Python. +Indexes are built and maintained through the Hudi metadata table (MDT) +indexing lifecycle. + +The search engine follows Lance's useful architectural choices without making +Lance a storage dependency: immutable segments, a compact term dictionary, +compressed posting lists, document-length statistics, BM25 ranking, positions, +and block-max WAND. Hudi owns the analyzer contract and on-disk format. Spark +integration remains in the main Hudi repository; Rust and Python implementation +work belongs in Hudi RS and consumes the same versioned format. + +The MDT remains authoritative for index definitions, visibility, coverage, +rollbacks, and cleaning. Large immutable posting payloads are sidecar files in +an auxiliary directory owned by the MDT rather than values embedded in HFiles. +An MDT commit atomically publishes descriptors for already durable payloads. + +Queries are snapshot-safe. SQL predicates always combine index results with a +raw scan of source file slices not covered by a compatible segment, so using an +index never changes SQL results. The direct search API also defaults to complete +results and may offer an explicitly incomplete low-latency mode. + +## Background + +Hudi indexes currently answer questions such as which files might contain a +record key or a value range. Full-text search has a different contract: analyze +free text into terms, locate matching documents, optionally verify positions, +and, for direct search APIs, rank the best documents. Sending this workload to +Elasticsearch or OpenSearch is effective, but creates a second ingestion +pipeline and a second source of snapshot and retention truth. + +This proposal builds on: + +- [RFC-45](../rfc-45/rfc-45.md), which introduced asynchronous MDT indexing; +- [RFC-77](../rfc-77/rfc-77.md), which established dynamically named secondary + index partitions and index definitions; +- the standard Spark SQL predicate model, which keeps index acceleration + transparent to relational queries; and +- RFC-109, the native vector-index proposal listed in the RFC catalog. Text and + vector search may share Hudi RS storage adapters and top-k utilities, but + their persistent formats remain independent. + +### Design principles + +1. The Hudi timeline is the source of snapshot truth. +2. Index creation, visibility, rollback, and cleaning use MDT components. +3. Immutable payloads support object-store range reads and safe caching. +4. Index use never changes the result of a Spark SQL predicate. +5. Ranking is independent of how the index is physically partitioned. +6. JVM and Hudi RS clients share query semantics and format versions. + +### Goals + +- Match token, boolean, prefix, fuzzy, phrase, and multi-column queries. +- Support copy-on-write (COW) and merge-on-read (MOR) tables. +- Build asynchronously and incrementally using the MDT indexer lifecycle. +- Guarantee snapshot-correct results for SQL and the default direct API mode. +- Provide an object-store-friendly text-index format with bounded memory usage. +- Provide a direct Hudi RS Python API alongside Spark SQL. + +### Non-goals + +- Elasticsearch API, aggregation, highlighting, or percolator compatibility. +- Highlighting and custom relevance models in the first format version. +- Updating posting lists in place. +- Replacing SQL predicate indexes or the record index. +- Adding Rust code or native build integration to the main Hudi repository. + +### Alternatives considered + +**External Elasticsearch/OpenSearch.** This remains a valid integration, but it +requires change-data-capture coordination, separate retention, and explicit +mapping between external documents and a Hudi snapshot. + +**Embedding Tantivy.** Tantivy is mature and Rust-native. Its archive and +directory abstractions, however, become a second persistent compatibility +contract. A smaller Hudi-owned format gives the project control over source +file-slice identity, range-read layout, and MDT publication semantics. + +**Posting lists as MDT record values.** This would make MDT storage atomic, but +multi-gigabyte postings, merges, and random term reads fight the metadata +table's record-oriented HFile/MOR strengths. Small authoritative descriptors in +the MDT plus immutable sidecars preserve the lifecycle benefits without that +cost. + +## Implementation + +### Terminology + +| Term | Meaning | +| --- | --- | +| Logical index | A named SQL index and its immutable analyzer configuration. | +| MDT index partition | Dynamic `text_index_<name>` metadata partition containing authoritative control records. | +| Segment | An immutable set of documents and term postings built together. | +| Payload partition | A shard within a segment whose local document identifiers are `u32`. | +| Source slice | A Hudi base file and its ordered log files at a snapshot. | +| Coverage | Proof that a segment represents a particular source-slice fingerprint. | +| Raw tail | Eligible source slices not covered by compatible visible segments. | +| Document | One Hudi record, identified by record key and source information. | + +### SQL interface + +Creation follows Hudi's secondary-index syntax: + +```sql +CREATE INDEX article_body_fts +ON articles +USING text_index (body) +OPTIONS ( + 'base_tokenizer' = 'simple', + 'lower_case' = 'true', + 'language' = 'und', + 'with_position' = 'true', + 'posting_block_size' = '128' +); +``` + +The existing `HoodieIndexDefinition` is populated as follows: + +```json +{ + "indexName": "article_body_fts", + "indexType": "text_index", + "sourceFields": ["body"], + "indexFunction": "tokenize", + "indexOptions": { + "base_tokenizer": "simple", + "lower_case": "true", + "language": "und", + "with_position": "true", + "posting_block_size": "128" + } +} +``` + +#### Spark SQL predicates + +The primary Spark interface follows the conventional index experience: a +boolean predicate in `WHERE`. Whether the optimizer uses the text index is not +observable in query results. + +```sql +SELECT _hoodie_record_key, title +FROM articles +WHERE hudi_match(body, 'lakehouse indexing', 'operator=AND'); +``` + +Phrase and multi-column queries use companion predicates: + +```sql +SELECT _hoodie_record_key, title +FROM articles +WHERE hudi_match_phrase(body, 'metadata table', 1) + AND category = 'engineering'; + +SELECT _hoodie_record_key, title +FROM articles +WHERE hudi_multi_match('lakehouse indexing', title, body); +``` + +The v1 signatures are: + +```text +hudi_match(column, query [, 'key=value,...']) -> boolean +hudi_match_phrase(column, query [, slop]) -> boolean +hudi_multi_match(query [, 'operator=AND|OR'], column, ...) -> boolean +``` + +`hudi_match` options include `operator`, `fuzziness`, `prefix_length`, and +`max_expansions`. Unknown options are rejected. The functions compose with +normal SQL predicates. Conjunctive text predicates can be pushed into index +planning; expressions whose `OR` semantics cannot be preserved are evaluated +by Spark without index pushdown. + +SQL predicate evaluation is always complete. Compatible segments accelerate +covered source slices, while uncovered or incompatible slices are evaluated by +the normal Hudi scan. There is no `_score` column and `LIMIT` does not imply +relevance order. Ranked top-k is a separate direct-search contract. + +#### Hudi RS Python table API + +Python users should not need to construct Spark SQL strings. Hudi RS extends +its existing `HudiTableBuilder` and `read_snapshot` API with the same structured +query model used by the index. The proposed predicate-style API is: + +```python +import pyarrow as pa + +from hudi import HudiTableBuilder +from hudi.search import FullTextOperator, MatchQuery, PhraseQuery + +table = ( + HudiTableBuilder + .from_base_uri("s3://warehouse/articles") + .build() +) + +query = MatchQuery( + "lakehouse indexing", + column="body", + operator=FullTextOperator.AND, +) + +batches = table.read_snapshot( + columns=["_hoodie_record_key", "title", "body"], + filters=[("category", "=", "engineering")], + full_text_query=query, +) +articles = pa.Table.from_batches(batches) +``` + +Structured queries compose without inventing a second query language: + +```python +query = ( + MatchQuery("metadata table", column="body") + & PhraseQuery("incremental indexing", column="body", slop=1) +) + +batches = table.read_snapshot(full_text_query=query) +``` + +For search applications, a Lance-style fluent API exposes ranked top-k and an +explicit score: + +```python +results = ( + table.search_text( + MatchQuery("lakehouse indexing", column="body"), + ) + .where([("category", "=", "engineering")]) + .select(["_hoodie_record_key", "title", "body"]) + .limit(20) + .to_arrow() +) + +# Ranked by BM25 descending; `_score` is included in `results`. +``` + +`read_snapshot(full_text_query=...)` has predicate semantics and returns every +match. `search_text(...).limit(k)` has ranked-search semantics and returns +`_score`. Both pin one Hudi snapshot, use identical analyzer/query objects, and +raw-scan uncovered source slices by default. The fluent API may expose +`allow_incomplete_index=True`, but it must mark the result metadata as +incomplete rather than silently changing defaults. + +Index creation remains an MDT table-service operation in the initial release, +invoked through Spark SQL or the Java API. A future Hudi RS writer API may add +`create_text_index` only after it can publish the corresponding MDT timeline +changes safely. + +The three query entry points share one query model and coverage planner, but +their result contracts differ: + +```mermaid +flowchart LR + SQL["Spark SQL<br/>hudi_match(...)"] + SNAPSHOT["Hudi RS Python<br/>read_snapshot(full_text_query=...)"] + SEARCH["Hudi RS Python<br/>search_text(...).limit(k)"] + MODEL["Shared query objects,<br/>analyzer contract, and snapshot pinning"] + MATCHES["Complete unordered<br/>match set"] + RANKED["BM25-ranked top-k<br/>with _score"] + + SQL --> MODEL + SNAPSHOT --> MODEL + SEARCH --> MODEL + MODEL -->|"predicate semantics"| MATCHES + MODEL -->|"ranked-search semantics"| RANKED +``` + +### Architecture + +```mermaid +flowchart TD + TIMELINE["Hudi data timeline<br/>snapshot S"] + DEFINITION["Index definition<br/>.hoodie/.index/index.json"] + MDT["MDT partition<br/>text_index_<name>"] + CONTROL["HEAD, SEGMENT, COVERAGE,<br/>and TOMBSTONE records"] + SIDECARS["Immutable sidecars<br/>tokens, docs, postings, positions"] + PLANNER["Snapshot coverage planner"] + INDEXED["Exactly covered<br/>source slices"] + RAW["Changed or uncovered<br/>raw source slices"] + INDEX_SCAN["Posting-list and<br/>position evaluation"] + RAW_SCAN["Normal Hudi scan with<br/>the same analyzer/query"] + UNION["Union and deduplicate<br/>document addresses"] + ROWS["Materialize rows and<br/>apply residual filters"] + RANK["Optional Hudi RS<br/>global BM25 top-k"] + + TIMELINE --> PLANNER + DEFINITION --> MDT + MDT --> CONTROL + CONTROL --> SIDECARS + CONTROL --> PLANNER + PLANNER --> INDEXED + PLANNER --> RAW + INDEXED --> INDEX_SCAN + SIDECARS --> INDEX_SCAN + RAW --> RAW_SCAN + INDEX_SCAN --> UNION + RAW_SCAN --> UNION + UNION --> ROWS + UNION --> RANK +``` + +The SQL definition is stored in `.hoodie/.index/index.json`. A dynamic MDT +partition stores small control records. Immutable native payload files live +under an MDT-owned auxiliary namespace. Readers never infer visibility by +listing that namespace; they use descriptors visible in the pinned MDT +snapshot. + +### Metadata table integration + +Add `TEXT_INDEX` to `MetadataPartitionType` with the dynamic prefix +`text_index_`. `getPartitionPath(metaClient, indexName)` and index-definition +lookup follow the secondary and expression index conventions. Add a +`TextSearchIndexer` to `IndexerFactory`, implementing `BaseIndexer` lifecycle +operations. + +Add a tagged Avro metadata payload named `HoodieTextIndexInfo` to +`HoodieMetadata.avsc`. The record is deliberately descriptor-sized and has +four logical kinds: + +| Kind | Record key | Purpose | +| --- | --- | --- | +| `HEAD` | `head` | Format, analyzer fingerprint, publication instant, aggregate stats. | +| `SEGMENT` | `segment/<uuid>` | Payload paths, sizes, checksums, statistics, source-mask cardinality. | +| `COVERAGE` | `coverage/<encoded-partition>/<file-id>` | Source fingerprint and ordered candidate segments. | Review Comment: Addressed. Coverage record is now defined as an MDT record mapping one data table file group to candidate index segments and their exact file-slice identities. A segment is usable only when that identity exactly matches the file slice selected for the pinned snapshot. ########## rfc/rfc-full-text-search/rfc-full-text-search.md: ########## @@ -0,0 +1,891 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# RFC: Hudi Full-Text Search Index + +## Proposers + +- @danny0405 + +## Approvers + +- TBD + +## Status + +Issue: TBD + +> This proposal intentionally does not claim an RFC number. The number and +> catalog entry will be reserved through the separate RFC-number process. + +## Abstract + +This RFC proposes a full-text index for Apache Hudi. It supports token, phrase, +and multi-column search over string columns, exposes standard predicates through +Spark SQL, and offers a Lance-style direct table API through Hudi RS Python. +Indexes are built and maintained through the Hudi metadata table (MDT) +indexing lifecycle. + +The search engine follows Lance's useful architectural choices without making +Lance a storage dependency: immutable segments, a compact term dictionary, +compressed posting lists, document-length statistics, BM25 ranking, positions, +and block-max WAND. Hudi owns the analyzer contract and on-disk format. Spark +integration remains in the main Hudi repository; Rust and Python implementation +work belongs in Hudi RS and consumes the same versioned format. + +The MDT remains authoritative for index definitions, visibility, coverage, +rollbacks, and cleaning. Large immutable posting payloads are sidecar files in +an auxiliary directory owned by the MDT rather than values embedded in HFiles. +An MDT commit atomically publishes descriptors for already durable payloads. + +Queries are snapshot-safe. SQL predicates always combine index results with a +raw scan of source file slices not covered by a compatible segment, so using an +index never changes SQL results. The direct search API also defaults to complete +results and may offer an explicitly incomplete low-latency mode. + +## Background + +Hudi indexes currently answer questions such as which files might contain a +record key or a value range. Full-text search has a different contract: analyze +free text into terms, locate matching documents, optionally verify positions, +and, for direct search APIs, rank the best documents. Sending this workload to +Elasticsearch or OpenSearch is effective, but creates a second ingestion +pipeline and a second source of snapshot and retention truth. + +This proposal builds on: + +- [RFC-45](../rfc-45/rfc-45.md), which introduced asynchronous MDT indexing; +- [RFC-77](../rfc-77/rfc-77.md), which established dynamically named secondary + index partitions and index definitions; +- the standard Spark SQL predicate model, which keeps index acceleration + transparent to relational queries; and +- RFC-109, the native vector-index proposal listed in the RFC catalog. Text and + vector search may share Hudi RS storage adapters and top-k utilities, but + their persistent formats remain independent. + +### Design principles + +1. The Hudi timeline is the source of snapshot truth. +2. Index creation, visibility, rollback, and cleaning use MDT components. +3. Immutable payloads support object-store range reads and safe caching. +4. Index use never changes the result of a Spark SQL predicate. +5. Ranking is independent of how the index is physically partitioned. +6. JVM and Hudi RS clients share query semantics and format versions. + +### Goals + +- Match token, boolean, prefix, fuzzy, phrase, and multi-column queries. +- Support copy-on-write (COW) and merge-on-read (MOR) tables. +- Build asynchronously and incrementally using the MDT indexer lifecycle. +- Guarantee snapshot-correct results for SQL and the default direct API mode. +- Provide an object-store-friendly text-index format with bounded memory usage. +- Provide a direct Hudi RS Python API alongside Spark SQL. + +### Non-goals + +- Elasticsearch API, aggregation, highlighting, or percolator compatibility. +- Highlighting and custom relevance models in the first format version. +- Updating posting lists in place. +- Replacing SQL predicate indexes or the record index. +- Adding Rust code or native build integration to the main Hudi repository. + +### Alternatives considered + +**External Elasticsearch/OpenSearch.** This remains a valid integration, but it +requires change-data-capture coordination, separate retention, and explicit +mapping between external documents and a Hudi snapshot. + +**Embedding Tantivy.** Tantivy is mature and Rust-native. Its archive and +directory abstractions, however, become a second persistent compatibility +contract. A smaller Hudi-owned format gives the project control over source +file-slice identity, range-read layout, and MDT publication semantics. + +**Posting lists as MDT record values.** This would make MDT storage atomic, but +multi-gigabyte postings, merges, and random term reads fight the metadata +table's record-oriented HFile/MOR strengths. Small authoritative descriptors in +the MDT plus immutable sidecars preserve the lifecycle benefits without that +cost. + +## Implementation + +### Terminology + +| Term | Meaning | +| --- | --- | +| Logical index | A named SQL index and its immutable analyzer configuration. | +| MDT index partition | Dynamic `text_index_<name>` metadata partition containing authoritative control records. | +| Segment | An immutable set of documents and term postings built together. | +| Payload partition | A shard within a segment whose local document identifiers are `u32`. | +| Source slice | A Hudi base file and its ordered log files at a snapshot. | +| Coverage | Proof that a segment represents a particular source-slice fingerprint. | +| Raw tail | Eligible source slices not covered by compatible visible segments. | +| Document | One Hudi record, identified by record key and source information. | + +### SQL interface + +Creation follows Hudi's secondary-index syntax: + +```sql +CREATE INDEX article_body_fts +ON articles +USING text_index (body) +OPTIONS ( + 'base_tokenizer' = 'simple', + 'lower_case' = 'true', + 'language' = 'und', + 'with_position' = 'true', + 'posting_block_size' = '128' +); +``` + +The existing `HoodieIndexDefinition` is populated as follows: + +```json +{ + "indexName": "article_body_fts", + "indexType": "text_index", + "sourceFields": ["body"], + "indexFunction": "tokenize", + "indexOptions": { + "base_tokenizer": "simple", + "lower_case": "true", + "language": "und", + "with_position": "true", + "posting_block_size": "128" + } +} +``` + +#### Spark SQL predicates + +The primary Spark interface follows the conventional index experience: a +boolean predicate in `WHERE`. Whether the optimizer uses the text index is not +observable in query results. + +```sql +SELECT _hoodie_record_key, title +FROM articles +WHERE hudi_match(body, 'lakehouse indexing', 'operator=AND'); +``` + +Phrase and multi-column queries use companion predicates: + +```sql +SELECT _hoodie_record_key, title +FROM articles +WHERE hudi_match_phrase(body, 'metadata table', 1) + AND category = 'engineering'; + +SELECT _hoodie_record_key, title +FROM articles +WHERE hudi_multi_match('lakehouse indexing', title, body); +``` + +The v1 signatures are: + +```text +hudi_match(column, query [, 'key=value,...']) -> boolean +hudi_match_phrase(column, query [, slop]) -> boolean +hudi_multi_match(query [, 'operator=AND|OR'], column, ...) -> boolean +``` + +`hudi_match` options include `operator`, `fuzziness`, `prefix_length`, and +`max_expansions`. Unknown options are rejected. The functions compose with +normal SQL predicates. Conjunctive text predicates can be pushed into index +planning; expressions whose `OR` semantics cannot be preserved are evaluated +by Spark without index pushdown. + +SQL predicate evaluation is always complete. Compatible segments accelerate +covered source slices, while uncovered or incompatible slices are evaluated by +the normal Hudi scan. There is no `_score` column and `LIMIT` does not imply +relevance order. Ranked top-k is a separate direct-search contract. + +#### Hudi RS Python table API + +Python users should not need to construct Spark SQL strings. Hudi RS extends +its existing `HudiTableBuilder` and `read_snapshot` API with the same structured +query model used by the index. The proposed predicate-style API is: + +```python +import pyarrow as pa + +from hudi import HudiTableBuilder +from hudi.search import FullTextOperator, MatchQuery, PhraseQuery + +table = ( + HudiTableBuilder + .from_base_uri("s3://warehouse/articles") + .build() +) + +query = MatchQuery( + "lakehouse indexing", + column="body", + operator=FullTextOperator.AND, +) + +batches = table.read_snapshot( + columns=["_hoodie_record_key", "title", "body"], + filters=[("category", "=", "engineering")], + full_text_query=query, +) +articles = pa.Table.from_batches(batches) +``` + +Structured queries compose without inventing a second query language: + +```python +query = ( + MatchQuery("metadata table", column="body") + & PhraseQuery("incremental indexing", column="body", slop=1) +) + +batches = table.read_snapshot(full_text_query=query) +``` + +For search applications, a Lance-style fluent API exposes ranked top-k and an +explicit score: + +```python +results = ( + table.search_text( + MatchQuery("lakehouse indexing", column="body"), + ) + .where([("category", "=", "engineering")]) + .select(["_hoodie_record_key", "title", "body"]) + .limit(20) + .to_arrow() +) + +# Ranked by BM25 descending; `_score` is included in `results`. +``` + +`read_snapshot(full_text_query=...)` has predicate semantics and returns every +match. `search_text(...).limit(k)` has ranked-search semantics and returns +`_score`. Both pin one Hudi snapshot, use identical analyzer/query objects, and +raw-scan uncovered source slices by default. The fluent API may expose +`allow_incomplete_index=True`, but it must mark the result metadata as +incomplete rather than silently changing defaults. + +Index creation remains an MDT table-service operation in the initial release, +invoked through Spark SQL or the Java API. A future Hudi RS writer API may add +`create_text_index` only after it can publish the corresponding MDT timeline +changes safely. + +The three query entry points share one query model and coverage planner, but +their result contracts differ: + +```mermaid +flowchart LR + SQL["Spark SQL<br/>hudi_match(...)"] + SNAPSHOT["Hudi RS Python<br/>read_snapshot(full_text_query=...)"] + SEARCH["Hudi RS Python<br/>search_text(...).limit(k)"] + MODEL["Shared query objects,<br/>analyzer contract, and snapshot pinning"] + MATCHES["Complete unordered<br/>match set"] + RANKED["BM25-ranked top-k<br/>with _score"] + + SQL --> MODEL + SNAPSHOT --> MODEL + SEARCH --> MODEL + MODEL -->|"predicate semantics"| MATCHES + MODEL -->|"ranked-search semantics"| RANKED +``` + +### Architecture + +```mermaid +flowchart TD + TIMELINE["Hudi data timeline<br/>snapshot S"] + DEFINITION["Index definition<br/>.hoodie/.index/index.json"] + MDT["MDT partition<br/>text_index_<name>"] + CONTROL["HEAD, SEGMENT, COVERAGE,<br/>and TOMBSTONE records"] + SIDECARS["Immutable sidecars<br/>tokens, docs, postings, positions"] + PLANNER["Snapshot coverage planner"] + INDEXED["Exactly covered<br/>source slices"] + RAW["Changed or uncovered<br/>raw source slices"] + INDEX_SCAN["Posting-list and<br/>position evaluation"] + RAW_SCAN["Normal Hudi scan with<br/>the same analyzer/query"] + UNION["Union and deduplicate<br/>document addresses"] + ROWS["Materialize rows and<br/>apply residual filters"] + RANK["Optional Hudi RS<br/>global BM25 top-k"] + + TIMELINE --> PLANNER + DEFINITION --> MDT + MDT --> CONTROL + CONTROL --> SIDECARS + CONTROL --> PLANNER + PLANNER --> INDEXED + PLANNER --> RAW + INDEXED --> INDEX_SCAN + SIDECARS --> INDEX_SCAN + RAW --> RAW_SCAN + INDEX_SCAN --> UNION + RAW_SCAN --> UNION + UNION --> ROWS + UNION --> RANK +``` + +The SQL definition is stored in `.hoodie/.index/index.json`. A dynamic MDT +partition stores small control records. Immutable native payload files live +under an MDT-owned auxiliary namespace. Readers never infer visibility by +listing that namespace; they use descriptors visible in the pinned MDT +snapshot. + +### Metadata table integration + +Add `TEXT_INDEX` to `MetadataPartitionType` with the dynamic prefix +`text_index_`. `getPartitionPath(metaClient, indexName)` and index-definition +lookup follow the secondary and expression index conventions. Add a +`TextSearchIndexer` to `IndexerFactory`, implementing `BaseIndexer` lifecycle +operations. + +Add a tagged Avro metadata payload named `HoodieTextIndexInfo` to +`HoodieMetadata.avsc`. The record is deliberately descriptor-sized and has +four logical kinds: + +| Kind | Record key | Purpose | +| --- | --- | --- | +| `HEAD` | `head` | Format, analyzer fingerprint, publication instant, aggregate stats. | +| `SEGMENT` | `segment/<uuid>` | Payload paths, sizes, checksums, statistics, source-mask cardinality. | +| `COVERAGE` | `coverage/<encoded-partition>/<file-id>` | Source fingerprint and ordered candidate segments. | +| `TOMBSTONE` | `tombstone/<uuid>` | Segment retirement instant and deletion eligibility. | + +The record includes a schema version, index name, analyzer fingerprint, segment +UUID, payload format version, source fingerprints, source-ordinal mapping, +aggregate document and token counts, file descriptors, and optional tombstone +instant. Large term statistics, dictionaries, and postings are never placed in +the Avro record. Active-source masks are computed for the pinned snapshot rather +than persisted as a single current value. + +The coverage planner does not scan the complete `text_index_<name>` partition +for every query. It first applies normal partition pruning and enumerates the +eligible data file slices, then issues batched point lookups for their +`coverage/<encoded-partition>/<file-id>` keys. It loads only the `SEGMENT` +descriptors referenced by those records and may cache immutable descriptors for +the lifetime of the pinned MDT snapshot. Coverage storage is `O(F_table)` in the +number of table file groups, while lookup and comparison work is `O(F_query)` in +the number of file groups selected by the query. An unpartitioned full-table +query still has `F_query = F_table`, consistent with its data-scan planning +scope. The dynamic MDT partition uses normal MDT file-group sharding, +compaction, and key lookup rather than a driver-side enumeration of all control +records. + +Payloads use this default path: + +```text +<table>/.hoodie/metadata/.aux/text-index/ + <escaped-index-name>/<segment-uuid>/... +``` + +The directory is below MDT ownership but outside normal MOR partition discovery. +A future external payload tier may be configured, but every path must be scoped +by table UUID and validated by readers. Only an MDT commit makes a segment +visible. Failed writers may leave unpublished files; the cleaner removes them +after a safety interval. + +### Source identity and snapshot coverage + +Each document stores this logical address: Review Comment: Addressed. The section was rewritten as Hudi record identity and file-slice coverage. The indexed unit is a merged Hudi table record located by record key and data table file slice; the format stores a file-slice ordinal, record key, and optional row-position hint rather than an independent search document identity. ########## rfc/rfc-full-text-search/rfc-full-text-search.md: ########## @@ -0,0 +1,891 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# RFC: Hudi Full-Text Search Index + +## Proposers + +- @danny0405 + +## Approvers + +- TBD + +## Status + +Issue: TBD + +> This proposal intentionally does not claim an RFC number. The number and +> catalog entry will be reserved through the separate RFC-number process. + +## Abstract + +This RFC proposes a full-text index for Apache Hudi. It supports token, phrase, +and multi-column search over string columns, exposes standard predicates through +Spark SQL, and offers a Lance-style direct table API through Hudi RS Python. +Indexes are built and maintained through the Hudi metadata table (MDT) +indexing lifecycle. + +The search engine follows Lance's useful architectural choices without making +Lance a storage dependency: immutable segments, a compact term dictionary, +compressed posting lists, document-length statistics, BM25 ranking, positions, +and block-max WAND. Hudi owns the analyzer contract and on-disk format. Spark +integration remains in the main Hudi repository; Rust and Python implementation +work belongs in Hudi RS and consumes the same versioned format. + +The MDT remains authoritative for index definitions, visibility, coverage, +rollbacks, and cleaning. Large immutable posting payloads are sidecar files in +an auxiliary directory owned by the MDT rather than values embedded in HFiles. +An MDT commit atomically publishes descriptors for already durable payloads. + +Queries are snapshot-safe. SQL predicates always combine index results with a +raw scan of source file slices not covered by a compatible segment, so using an +index never changes SQL results. The direct search API also defaults to complete +results and may offer an explicitly incomplete low-latency mode. + +## Background + +Hudi indexes currently answer questions such as which files might contain a +record key or a value range. Full-text search has a different contract: analyze +free text into terms, locate matching documents, optionally verify positions, +and, for direct search APIs, rank the best documents. Sending this workload to +Elasticsearch or OpenSearch is effective, but creates a second ingestion +pipeline and a second source of snapshot and retention truth. + +This proposal builds on: + +- [RFC-45](../rfc-45/rfc-45.md), which introduced asynchronous MDT indexing; +- [RFC-77](../rfc-77/rfc-77.md), which established dynamically named secondary + index partitions and index definitions; +- the standard Spark SQL predicate model, which keeps index acceleration + transparent to relational queries; and +- RFC-109, the native vector-index proposal listed in the RFC catalog. Text and + vector search may share Hudi RS storage adapters and top-k utilities, but + their persistent formats remain independent. + +### Design principles + +1. The Hudi timeline is the source of snapshot truth. +2. Index creation, visibility, rollback, and cleaning use MDT components. +3. Immutable payloads support object-store range reads and safe caching. +4. Index use never changes the result of a Spark SQL predicate. +5. Ranking is independent of how the index is physically partitioned. +6. JVM and Hudi RS clients share query semantics and format versions. + +### Goals + +- Match token, boolean, prefix, fuzzy, phrase, and multi-column queries. +- Support copy-on-write (COW) and merge-on-read (MOR) tables. +- Build asynchronously and incrementally using the MDT indexer lifecycle. +- Guarantee snapshot-correct results for SQL and the default direct API mode. +- Provide an object-store-friendly text-index format with bounded memory usage. +- Provide a direct Hudi RS Python API alongside Spark SQL. + +### Non-goals + +- Elasticsearch API, aggregation, highlighting, or percolator compatibility. +- Highlighting and custom relevance models in the first format version. +- Updating posting lists in place. +- Replacing SQL predicate indexes or the record index. +- Adding Rust code or native build integration to the main Hudi repository. + +### Alternatives considered + +**External Elasticsearch/OpenSearch.** This remains a valid integration, but it +requires change-data-capture coordination, separate retention, and explicit +mapping between external documents and a Hudi snapshot. + +**Embedding Tantivy.** Tantivy is mature and Rust-native. Its archive and +directory abstractions, however, become a second persistent compatibility +contract. A smaller Hudi-owned format gives the project control over source +file-slice identity, range-read layout, and MDT publication semantics. + +**Posting lists as MDT record values.** This would make MDT storage atomic, but +multi-gigabyte postings, merges, and random term reads fight the metadata +table's record-oriented HFile/MOR strengths. Small authoritative descriptors in +the MDT plus immutable sidecars preserve the lifecycle benefits without that +cost. + +## Implementation + +### Terminology + +| Term | Meaning | +| --- | --- | +| Logical index | A named SQL index and its immutable analyzer configuration. | +| MDT index partition | Dynamic `text_index_<name>` metadata partition containing authoritative control records. | +| Segment | An immutable set of documents and term postings built together. | +| Payload partition | A shard within a segment whose local document identifiers are `u32`. | +| Source slice | A Hudi base file and its ordered log files at a snapshot. | +| Coverage | Proof that a segment represents a particular source-slice fingerprint. | +| Raw tail | Eligible source slices not covered by compatible visible segments. | +| Document | One Hudi record, identified by record key and source information. | + +### SQL interface + +Creation follows Hudi's secondary-index syntax: + +```sql +CREATE INDEX article_body_fts +ON articles +USING text_index (body) +OPTIONS ( + 'base_tokenizer' = 'simple', + 'lower_case' = 'true', + 'language' = 'und', + 'with_position' = 'true', + 'posting_block_size' = '128' +); +``` + +The existing `HoodieIndexDefinition` is populated as follows: + +```json +{ + "indexName": "article_body_fts", + "indexType": "text_index", + "sourceFields": ["body"], + "indexFunction": "tokenize", + "indexOptions": { + "base_tokenizer": "simple", + "lower_case": "true", + "language": "und", + "with_position": "true", + "posting_block_size": "128" + } +} +``` + +#### Spark SQL predicates + +The primary Spark interface follows the conventional index experience: a +boolean predicate in `WHERE`. Whether the optimizer uses the text index is not +observable in query results. + +```sql +SELECT _hoodie_record_key, title +FROM articles +WHERE hudi_match(body, 'lakehouse indexing', 'operator=AND'); +``` + +Phrase and multi-column queries use companion predicates: + +```sql +SELECT _hoodie_record_key, title +FROM articles +WHERE hudi_match_phrase(body, 'metadata table', 1) + AND category = 'engineering'; + +SELECT _hoodie_record_key, title +FROM articles +WHERE hudi_multi_match('lakehouse indexing', title, body); +``` + +The v1 signatures are: + +```text +hudi_match(column, query [, 'key=value,...']) -> boolean +hudi_match_phrase(column, query [, slop]) -> boolean +hudi_multi_match(query [, 'operator=AND|OR'], column, ...) -> boolean +``` + +`hudi_match` options include `operator`, `fuzziness`, `prefix_length`, and +`max_expansions`. Unknown options are rejected. The functions compose with +normal SQL predicates. Conjunctive text predicates can be pushed into index +planning; expressions whose `OR` semantics cannot be preserved are evaluated +by Spark without index pushdown. + +SQL predicate evaluation is always complete. Compatible segments accelerate +covered source slices, while uncovered or incompatible slices are evaluated by +the normal Hudi scan. There is no `_score` column and `LIMIT` does not imply +relevance order. Ranked top-k is a separate direct-search contract. + +#### Hudi RS Python table API + +Python users should not need to construct Spark SQL strings. Hudi RS extends +its existing `HudiTableBuilder` and `read_snapshot` API with the same structured +query model used by the index. The proposed predicate-style API is: + +```python +import pyarrow as pa + +from hudi import HudiTableBuilder +from hudi.search import FullTextOperator, MatchQuery, PhraseQuery + +table = ( + HudiTableBuilder + .from_base_uri("s3://warehouse/articles") + .build() +) + +query = MatchQuery( + "lakehouse indexing", + column="body", + operator=FullTextOperator.AND, +) + +batches = table.read_snapshot( + columns=["_hoodie_record_key", "title", "body"], + filters=[("category", "=", "engineering")], + full_text_query=query, +) +articles = pa.Table.from_batches(batches) +``` + +Structured queries compose without inventing a second query language: + +```python +query = ( + MatchQuery("metadata table", column="body") + & PhraseQuery("incremental indexing", column="body", slop=1) +) + +batches = table.read_snapshot(full_text_query=query) +``` + +For search applications, a Lance-style fluent API exposes ranked top-k and an +explicit score: + +```python +results = ( + table.search_text( + MatchQuery("lakehouse indexing", column="body"), + ) + .where([("category", "=", "engineering")]) + .select(["_hoodie_record_key", "title", "body"]) + .limit(20) + .to_arrow() +) + +# Ranked by BM25 descending; `_score` is included in `results`. +``` + +`read_snapshot(full_text_query=...)` has predicate semantics and returns every +match. `search_text(...).limit(k)` has ranked-search semantics and returns +`_score`. Both pin one Hudi snapshot, use identical analyzer/query objects, and +raw-scan uncovered source slices by default. The fluent API may expose +`allow_incomplete_index=True`, but it must mark the result metadata as +incomplete rather than silently changing defaults. + +Index creation remains an MDT table-service operation in the initial release, +invoked through Spark SQL or the Java API. A future Hudi RS writer API may add +`create_text_index` only after it can publish the corresponding MDT timeline +changes safely. + +The three query entry points share one query model and coverage planner, but +their result contracts differ: + +```mermaid +flowchart LR + SQL["Spark SQL<br/>hudi_match(...)"] + SNAPSHOT["Hudi RS Python<br/>read_snapshot(full_text_query=...)"] + SEARCH["Hudi RS Python<br/>search_text(...).limit(k)"] + MODEL["Shared query objects,<br/>analyzer contract, and snapshot pinning"] + MATCHES["Complete unordered<br/>match set"] + RANKED["BM25-ranked top-k<br/>with _score"] + + SQL --> MODEL + SNAPSHOT --> MODEL + SEARCH --> MODEL + MODEL -->|"predicate semantics"| MATCHES + MODEL -->|"ranked-search semantics"| RANKED +``` + +### Architecture + +```mermaid +flowchart TD + TIMELINE["Hudi data timeline<br/>snapshot S"] + DEFINITION["Index definition<br/>.hoodie/.index/index.json"] + MDT["MDT partition<br/>text_index_<name>"] + CONTROL["HEAD, SEGMENT, COVERAGE,<br/>and TOMBSTONE records"] + SIDECARS["Immutable sidecars<br/>tokens, docs, postings, positions"] + PLANNER["Snapshot coverage planner"] + INDEXED["Exactly covered<br/>source slices"] + RAW["Changed or uncovered<br/>raw source slices"] + INDEX_SCAN["Posting-list and<br/>position evaluation"] + RAW_SCAN["Normal Hudi scan with<br/>the same analyzer/query"] + UNION["Union and deduplicate<br/>document addresses"] + ROWS["Materialize rows and<br/>apply residual filters"] + RANK["Optional Hudi RS<br/>global BM25 top-k"] + + TIMELINE --> PLANNER + DEFINITION --> MDT + MDT --> CONTROL + CONTROL --> SIDECARS + CONTROL --> PLANNER + PLANNER --> INDEXED + PLANNER --> RAW + INDEXED --> INDEX_SCAN + SIDECARS --> INDEX_SCAN + RAW --> RAW_SCAN + INDEX_SCAN --> UNION + RAW_SCAN --> UNION + UNION --> ROWS + UNION --> RANK +``` + +The SQL definition is stored in `.hoodie/.index/index.json`. A dynamic MDT +partition stores small control records. Immutable native payload files live +under an MDT-owned auxiliary namespace. Readers never infer visibility by +listing that namespace; they use descriptors visible in the pinned MDT +snapshot. + +### Metadata table integration + +Add `TEXT_INDEX` to `MetadataPartitionType` with the dynamic prefix +`text_index_`. `getPartitionPath(metaClient, indexName)` and index-definition +lookup follow the secondary and expression index conventions. Add a +`TextSearchIndexer` to `IndexerFactory`, implementing `BaseIndexer` lifecycle +operations. + +Add a tagged Avro metadata payload named `HoodieTextIndexInfo` to +`HoodieMetadata.avsc`. The record is deliberately descriptor-sized and has +four logical kinds: + +| Kind | Record key | Purpose | +| --- | --- | --- | +| `HEAD` | `head` | Format, analyzer fingerprint, publication instant, aggregate stats. | +| `SEGMENT` | `segment/<uuid>` | Payload paths, sizes, checksums, statistics, source-mask cardinality. | +| `COVERAGE` | `coverage/<encoded-partition>/<file-id>` | Source fingerprint and ordered candidate segments. | +| `TOMBSTONE` | `tombstone/<uuid>` | Segment retirement instant and deletion eligibility. | + +The record includes a schema version, index name, analyzer fingerprint, segment +UUID, payload format version, source fingerprints, source-ordinal mapping, +aggregate document and token counts, file descriptors, and optional tombstone +instant. Large term statistics, dictionaries, and postings are never placed in +the Avro record. Active-source masks are computed for the pinned snapshot rather +than persisted as a single current value. + +The coverage planner does not scan the complete `text_index_<name>` partition +for every query. It first applies normal partition pruning and enumerates the +eligible data file slices, then issues batched point lookups for their +`coverage/<encoded-partition>/<file-id>` keys. It loads only the `SEGMENT` +descriptors referenced by those records and may cache immutable descriptors for +the lifetime of the pinned MDT snapshot. Coverage storage is `O(F_table)` in the +number of table file groups, while lookup and comparison work is `O(F_query)` in +the number of file groups selected by the query. An unpartitioned full-table +query still has `F_query = F_table`, consistent with its data-scan planning +scope. The dynamic MDT partition uses normal MDT file-group sharding, +compaction, and key lookup rather than a driver-side enumeration of all control +records. + +Payloads use this default path: + +```text +<table>/.hoodie/metadata/.aux/text-index/ + <escaped-index-name>/<segment-uuid>/... +``` + +The directory is below MDT ownership but outside normal MOR partition discovery. +A future external payload tier may be configured, but every path must be scoped +by table UUID and validated by readers. Only an MDT commit makes a segment +visible. Failed writers may leave unpublished files; the cleaner removes them +after a safety interval. + +### Source identity and snapshot coverage + +Each document stores this logical address: + +```text +HudiDocumentAddress { + source_ordinal: u32, + partition_path: bytes, + file_id: bytes, + record_key: bytes, + row_position_hint: optional u64 +} +``` + +Version 1 requires a stable Hudi record key. `file_id` lets materialization group +lookups, while `row_position_hint` is only an optimization and is used when the +source fingerprint matches exactly. + +A source fingerprint includes: + +- table UUID, partition path, and file ID; +- base instant and base-file identity (path, length, and checksum when known); +- ordered log-file identities (path, length, and latest block instant); +- writer schema identifier; and +- record-merger implementation and relevant options. + +At snapshot `S`, the coverage planner enumerates eligible file slices and +compares exact fingerprints. Exact matches activate the segment's source mask. +A changed base file or added MOR log makes that source uncovered until it is +rebuilt. A segment created from a later state is not used for an older snapshot. +This avoids attempting to delete or mutate old postings after compaction, +clustering, rollback, or MOR updates. + +Freshness is measured in changed source slices, not merely elapsed commits. Let +`F` be the eligible source slices and `R` the slices whose fingerprint is not +covered at the query snapshot. The coverage ratio is `C = 1 - R/F`. Once a MOR +file group receives its first new log block it contributes one raw slice until +catch-up; additional blocks increase scan bytes but not the raw-slice count. A +complete predicate query therefore has the qualitative cost +`index_scan(C * F) + raw_scan(R)`. Complete ranked search additionally analyzes +the raw tail to obtain exact global document frequencies. As `C` approaches +zero, performance intentionally approaches a normal Hudi scan while correctness +is unchanged. + +There is no universal freshness SLA because `R`, log size, analyzer cost, and +query selectivity depend on the workload. Deployments schedule incremental +catch-up by time or changed-slice thresholds and observe source instant lag, +coverage ratio, raw bytes, and raw-tail analysis time. The performance plan must +publish the latency envelope across those dimensions before the feature is +enabled by default. + +### Text-index payload format + +All files begin with an eight-byte `HUDIFTS1` magic value followed by little- +endian format version, feature flags, variable-header length, and header +checksum. Independently checksummed blocks follow the header, and a footer +contains the block directory for range reads. Readers reject unknown required +feature bits and enforce configured allocation limits before reading lengths. + +Each segment contains: + +- `metadata.hfts`: analyzer fingerprint, source table, source descriptors, + segment statistics, payload partition descriptors, and checksums; +- `part_<n>.tokens.hfts`: a minimal finite-state transducer (FST) mapping + analyzed term bytes to term ordinals and posting metadata; +- `part_<n>.docs.hfts`: columnar source ordinal, record-key offsets and bytes, + document token count, and optional row-position hint; +- `part_<n>.postings.hfts`: document frequency, posting-block offsets, + delta-encoded local document IDs, term frequencies, and block-max metadata; + and +- `part_<n>.positions.hfts`: optional delta-encoded token positions and offsets. + +Payload partitions use local `u32` document IDs. A builder starts a new payload +partition before that space is exhausted. Posting blocks default to 128 +documents and use bit packing or variable-byte encoding, selected per block. +Each block records maximum term frequency and minimum document length; these +values provide a conservative BM25 upper bound for block-max WAND. Positions +are stored only when enabled by the immutable index definition. + +A phrase clause requires compatible position data. For an index created with +`with_position=false`, the complete Spark SQL and Hudi RS paths treat its source +slices as uncovered for that query and evaluate the phrase through +`RawTextSearchSplit`. They never silently downgrade a phrase to an `AND` of its +terms. In version 1, `allow_incomplete_index=True` rejects a phrase query against +a positionless index rather than returning an approximate result. + +No implementation-specific collection serialization is persisted directly. +Every field is defined by the Hudi format specification, so upgrading a Java or +Rust dependency cannot silently change files. + +### Implementation ownership + +The main Hudi repository owns the SQL extension, MDT index lifecycle, Spark +planning, and the language-neutral persistent format. It does not add an +in-tree Rust crate or JNI build for this feature. + +Any Rust reader, builder, tokenizer, or Python binding is developed and released +from Hudi RS. Hudi RS already owns Hudi's Rust implementation and Python +bindings, so this keeps native code, packaging, and Python API compatibility in +the appropriate project. The two repositories coordinate through versioned +contracts rather than source-code coupling: + +```text +Main Hudi repository Hudi RS repository +-------------------- ------------------ +Spark SQL predicates Python query objects +MDT indexer lifecycle read_snapshot(full_text_query=...) +HoodieIndexDefinition search_text(...) fluent API +HoodieTextIndexInfo Avro schema payload builder/reader implementation Review Comment: Addressed. The ownership table and Phase 1 rollout now explicitly assign a pure-Java index-file builder, reader, and analyzer conformance suite to the main Hudi repository. Phase 1 therefore builds and reads real indexed data in Spark; Hudi RS later supplies an independent Rust implementation and is not a Spark runtime dependency. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
