hudi-agent commented on code in PR #19613: URL: https://github.com/apache/hudi/pull/19613#discussion_r3771757495
########## rfc/rfc-110/rfc-110.md: ########## @@ -0,0 +1,639 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# RFC-110: Native Full-Text Search Index + +## Proposers + +- @danny0405 + +## Approvers + +- TBD + +## Status + +Issue: TBD + +> The RFC number is provisional until the community assigns an issue and +> accepts the proposal. RFC-109 is the highest numbered proposal in this +> checkout, so this draft uses RFC-110 to make repository review practical. + +## Abstract + +This RFC proposes a native, relevance-ranked full-text index for Apache Hudi. +It supports token and phrase search over string columns, exposes search through +a Spark table-valued function (TVF), and builds and maintains indexes through +the Hudi metadata table (MDT) indexing lifecycle. + +The search engine follows Lance's useful architectural choices without making +Lance a storage dependency: immutable segments, a compact term dictionary, +compressed posting lists, document-length statistics, BM25 ranking, positions, +and block-max WAND. Hudi owns the analyzer contract and on-disk format. The +format and hot search path are implemented in a Rust crate kept in the Hudi +repository and called through a narrow Java native boundary. + +The MDT remains authoritative for index definitions, visibility, coverage, +rollbacks, and cleaning. Large immutable posting payloads are sidecar files in +an auxiliary directory owned by the MDT rather than values embedded in HFiles. +An MDT commit atomically publishes descriptors for already durable payloads. + +Queries are snapshot-safe. The default `complete` mode combines native index +results with a raw scan of source file slices not covered by a compatible +segment. An opt-in `fast` mode searches only covered data and reports that it +may omit matches. + +## Background + +Hudi indexes currently answer questions such as which files might contain a +record key or a value range. Full-text search has a different contract: analyze +free text into terms, locate matching documents, optionally verify positions, +and rank the best documents. Sending this workload to Elasticsearch or +OpenSearch is effective, but creates a second ingestion pipeline and a second +source of snapshot and retention truth. + +This proposal builds on: + +- [RFC-45](../rfc-45/rfc-45.md), which introduced asynchronous MDT indexing; +- [RFC-77](../rfc-77/rfc-77.md), which established dynamically named secondary + index partitions and index definitions; +- [RFC-102](../rfc-102/rfc-102.md), whose vector-search TVF provides a useful + SQL precedent; and +- RFC-109, the native vector-index proposal listed in the RFC catalog. Text and + vector search should eventually share native artifact packaging, storage + adapters, and top-k execution utilities, but their persistent formats remain + independent. + +### Design principles + +1. The Hudi timeline is the source of snapshot truth. +2. Index creation, visibility, rollback, and cleaning use MDT components. +3. Immutable payloads support object-store range reads and safe caching. +4. Ranking is independent of how the index is physically partitioned. +5. Freshness and incompleteness are explicit query properties. +6. The Java/native interface is small enough to replace without changing SQL. + +### Goals + +- Rank token, boolean, prefix, fuzzy, and phrase queries on one string column. +- Support copy-on-write (COW) and merge-on-read (MOR) tables. +- Build asynchronously and incrementally using the MDT indexer lifecycle. +- Guarantee snapshot-correct results in the default search mode. +- Provide an object-store-friendly native format with bounded memory usage. +- Keep the initial Rust implementation inside the Hudi repository. + +### Non-goals + +- Elasticsearch API, aggregation, highlighting, or percolator compatibility. +- Multi-column relevance models in the first format version. +- Updating posting lists in place. +- Replacing SQL predicate indexes or the record index. +- A general Rust rewrite of Hudi readers. + +### Alternatives considered + +**External Elasticsearch/OpenSearch.** This remains a valid integration, but it +requires change-data-capture coordination, separate retention, and explicit +mapping between external documents and a Hudi snapshot. + +**Embedding Tantivy.** Tantivy is mature and Rust-native. Its archive and +directory abstractions, however, become a second persistent compatibility +contract. A smaller Hudi-owned format gives the project control over source +file-slice identity, range-read layout, and MDT publication semantics. + +**Posting lists as MDT record values.** This would make MDT storage atomic, but +multi-gigabyte postings, merges, and random term reads fight the metadata +table's record-oriented HFile/MOR strengths. Small authoritative descriptors in +the MDT plus immutable sidecars preserve the lifecycle benefits without that +cost. + +## Implementation + +### Terminology + +| Term | Meaning | +| --- | --- | +| Logical index | A named SQL index and its immutable analyzer configuration. | +| MDT index partition | Dynamic `text_index_<name>` metadata partition containing authoritative control records. | +| Segment | An immutable set of documents and term postings built together. | +| Payload partition | A shard within a segment whose local document identifiers are `u32`. | +| Source slice | A Hudi base file and its ordered log files at a snapshot. | +| Coverage | Proof that a segment represents a particular source-slice fingerprint. | +| Raw tail | Eligible source slices not covered by compatible visible segments. | +| Document | One Hudi record, identified by record key and source information. | + +### SQL interface + +Creation follows Hudi's secondary-index syntax: + +```sql +CREATE INDEX article_body_fts +ON articles +USING text_index (body) +OPTIONS ( + 'base_tokenizer' = 'simple', + 'lower_case' = 'true', + 'language' = 'und', + 'with_position' = 'true', + 'posting_block_size' = '128' +); +``` + +The existing `HoodieIndexDefinition` is populated as follows: + +```json +{ + "indexName": "article_body_fts", + "indexType": "text_index", + "sourceFields": ["body"], + "indexFunction": "tokenize", + "indexOptions": { + "base_tokenizer": "simple", + "lower_case": "true", + "language": "und", + "with_position": "true", + "posting_block_size": "128" + } Review Comment: 🤖 Search is exposed only as a Spark TVF, with Flink/Java deferred to Phase 3. Since the analyzer fingerprint and payload format are engine-neutral, could you clarify the cross-engine consistency contract — will a segment built by one engine be guaranteed identically searchable by another, and is the query DSL/scoring intended to be shared library code across engines rather than reimplemented per engine? <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub> ########## rfc/rfc-110/rfc-110.md: ########## @@ -0,0 +1,639 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# RFC-110: Native Full-Text Search Index + +## Proposers + +- @danny0405 + +## Approvers + +- TBD + +## Status + +Issue: TBD + +> The RFC number is provisional until the community assigns an issue and +> accepts the proposal. RFC-109 is the highest numbered proposal in this +> checkout, so this draft uses RFC-110 to make repository review practical. + +## Abstract + +This RFC proposes a native, relevance-ranked full-text index for Apache Hudi. +It supports token and phrase search over string columns, exposes search through +a Spark table-valued function (TVF), and builds and maintains indexes through +the Hudi metadata table (MDT) indexing lifecycle. + +The search engine follows Lance's useful architectural choices without making +Lance a storage dependency: immutable segments, a compact term dictionary, +compressed posting lists, document-length statistics, BM25 ranking, positions, +and block-max WAND. Hudi owns the analyzer contract and on-disk format. The +format and hot search path are implemented in a Rust crate kept in the Hudi +repository and called through a narrow Java native boundary. + +The MDT remains authoritative for index definitions, visibility, coverage, +rollbacks, and cleaning. Large immutable posting payloads are sidecar files in +an auxiliary directory owned by the MDT rather than values embedded in HFiles. +An MDT commit atomically publishes descriptors for already durable payloads. + +Queries are snapshot-safe. The default `complete` mode combines native index +results with a raw scan of source file slices not covered by a compatible +segment. An opt-in `fast` mode searches only covered data and reports that it +may omit matches. + +## Background + +Hudi indexes currently answer questions such as which files might contain a +record key or a value range. Full-text search has a different contract: analyze +free text into terms, locate matching documents, optionally verify positions, +and rank the best documents. Sending this workload to Elasticsearch or +OpenSearch is effective, but creates a second ingestion pipeline and a second +source of snapshot and retention truth. + +This proposal builds on: + +- [RFC-45](../rfc-45/rfc-45.md), which introduced asynchronous MDT indexing; +- [RFC-77](../rfc-77/rfc-77.md), which established dynamically named secondary + index partitions and index definitions; +- [RFC-102](../rfc-102/rfc-102.md), whose vector-search TVF provides a useful + SQL precedent; and +- RFC-109, the native vector-index proposal listed in the RFC catalog. Text and + vector search should eventually share native artifact packaging, storage + adapters, and top-k execution utilities, but their persistent formats remain + independent. + +### Design principles + +1. The Hudi timeline is the source of snapshot truth. +2. Index creation, visibility, rollback, and cleaning use MDT components. +3. Immutable payloads support object-store range reads and safe caching. +4. Ranking is independent of how the index is physically partitioned. +5. Freshness and incompleteness are explicit query properties. +6. The Java/native interface is small enough to replace without changing SQL. + +### Goals + +- Rank token, boolean, prefix, fuzzy, and phrase queries on one string column. +- Support copy-on-write (COW) and merge-on-read (MOR) tables. +- Build asynchronously and incrementally using the MDT indexer lifecycle. +- Guarantee snapshot-correct results in the default search mode. +- Provide an object-store-friendly native format with bounded memory usage. +- Keep the initial Rust implementation inside the Hudi repository. + +### Non-goals + +- Elasticsearch API, aggregation, highlighting, or percolator compatibility. +- Multi-column relevance models in the first format version. +- Updating posting lists in place. +- Replacing SQL predicate indexes or the record index. +- A general Rust rewrite of Hudi readers. + +### Alternatives considered + +**External Elasticsearch/OpenSearch.** This remains a valid integration, but it +requires change-data-capture coordination, separate retention, and explicit +mapping between external documents and a Hudi snapshot. + +**Embedding Tantivy.** Tantivy is mature and Rust-native. Its archive and +directory abstractions, however, become a second persistent compatibility +contract. A smaller Hudi-owned format gives the project control over source +file-slice identity, range-read layout, and MDT publication semantics. + +**Posting lists as MDT record values.** This would make MDT storage atomic, but +multi-gigabyte postings, merges, and random term reads fight the metadata +table's record-oriented HFile/MOR strengths. Small authoritative descriptors in +the MDT plus immutable sidecars preserve the lifecycle benefits without that +cost. + +## Implementation + +### Terminology + +| Term | Meaning | +| --- | --- | +| Logical index | A named SQL index and its immutable analyzer configuration. | +| MDT index partition | Dynamic `text_index_<name>` metadata partition containing authoritative control records. | +| Segment | An immutable set of documents and term postings built together. | +| Payload partition | A shard within a segment whose local document identifiers are `u32`. | +| Source slice | A Hudi base file and its ordered log files at a snapshot. | +| Coverage | Proof that a segment represents a particular source-slice fingerprint. | +| Raw tail | Eligible source slices not covered by compatible visible segments. | +| Document | One Hudi record, identified by record key and source information. | + +### SQL interface + +Creation follows Hudi's secondary-index syntax: + +```sql +CREATE INDEX article_body_fts +ON articles +USING text_index (body) +OPTIONS ( + 'base_tokenizer' = 'simple', + 'lower_case' = 'true', + 'language' = 'und', + 'with_position' = 'true', + 'posting_block_size' = '128' +); +``` + +The existing `HoodieIndexDefinition` is populated as follows: + +```json +{ + "indexName": "article_body_fts", + "indexType": "text_index", + "sourceFields": ["body"], + "indexFunction": "tokenize", + "indexOptions": { + "base_tokenizer": "simple", + "lower_case": "true", + "language": "und", + "with_position": "true", + "posting_block_size": "128" + } +} +``` + +Search is exposed as a Spark TVF rather than a boolean predicate because score +and top-k are part of its semantics: + +```sql +SELECT _hoodie_record_key, title, _score +FROM hudi_text_search( + table => 'articles', + column => 'body', + query => '{"match":{"query":"lakehouse indexing","operator":"and"}}', + k => 20, + options => map('search_mode', 'complete') +); +``` + +A phrase query is expressed as: + +```json +{"phrase":{"query":"metadata table","slop":1}} +``` + +The initial JSON query DSL contains `match`, `phrase`, `and`, `or`, and `not`. +Prefix and bounded edit-distance expansion are options on `match`. Unknown +fields and operators are rejected instead of ignored. Output includes the Hudi +record, `_score`, and `_hoodie_text_index` diagnostic metadata. Ties are ordered +by score descending, then partition path, record key, and file ID ascending. + +`search_mode` has three values: + +- `complete` (default): search compatible segments and raw-scan uncovered + source slices. Its result must equal a full raw search at the pinned snapshot. +- `fast`: search only compatible covered slices. The result metadata includes + coverage and sets `is_complete=false` when anything is skipped. +- `flat`: bypass the native index, useful as a correctness oracle and fallback. + +### Architecture + +```text + .hoodie/.index/index.json + | + v +Hudi timeline ---> text_index_<name> MDT partition ---> query coverage planner + | | | | + segment records coverage records | | + | | | + v v v +.hoodie/metadata/.aux/text-index/.../segment/ native splits raw splits + metadata.hfts part_*.tokens.hfts \ / + part_*.docs.hfts part_*.postings.hfts top-k merge + part_*.positions.hfts | + materialize rows +``` + +The SQL definition is stored in `.hoodie/.index/index.json`. A dynamic MDT +partition stores small control records. Immutable native payload files live +under an MDT-owned auxiliary namespace. Readers never infer visibility by +listing that namespace; they use descriptors visible in the pinned MDT +snapshot. + +### Metadata table integration + +Add `TEXT_INDEX` to `MetadataPartitionType` with the dynamic prefix +`text_index_`. `getPartitionPath(metaClient, indexName)` and index-definition +lookup follow the secondary and expression index conventions. Add a +`TextSearchIndexer` to `IndexerFactory`, implementing `BaseIndexer` lifecycle +operations. + +Add a tagged Avro metadata payload named `HoodieTextIndexInfo` to +`HoodieMetadata.avsc`. The record is deliberately descriptor-sized and has +four logical kinds: + +| Kind | Record key | Purpose | +| --- | --- | --- | +| `HEAD` | `head` | Format, analyzer fingerprint, publication instant, aggregate stats. | Review Comment: 🤖 How does async text-index building interact with concurrent data writers and the existing OCC/timeline concurrency control? Specifically, if a data commit lands between fingerprint enumeration (step 2) and MDT publication (step 10), what prevents publishing coverage for a slice that has already changed? @nsivabalan could you weigh in on whether the pin-instant + exact-fingerprint approach is sufficient under multi-writer, or if additional conflict handling against the MDT is needed here? <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub> ########## rfc/rfc-110/rfc-110.md: ########## @@ -0,0 +1,639 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# RFC-110: Native Full-Text Search Index + +## Proposers + +- @danny0405 + +## Approvers + +- TBD + +## Status + +Issue: TBD + +> The RFC number is provisional until the community assigns an issue and +> accepts the proposal. RFC-109 is the highest numbered proposal in this +> checkout, so this draft uses RFC-110 to make repository review practical. + +## Abstract + +This RFC proposes a native, relevance-ranked full-text index for Apache Hudi. +It supports token and phrase search over string columns, exposes search through +a Spark table-valued function (TVF), and builds and maintains indexes through +the Hudi metadata table (MDT) indexing lifecycle. + +The search engine follows Lance's useful architectural choices without making +Lance a storage dependency: immutable segments, a compact term dictionary, +compressed posting lists, document-length statistics, BM25 ranking, positions, +and block-max WAND. Hudi owns the analyzer contract and on-disk format. The +format and hot search path are implemented in a Rust crate kept in the Hudi +repository and called through a narrow Java native boundary. + +The MDT remains authoritative for index definitions, visibility, coverage, +rollbacks, and cleaning. Large immutable posting payloads are sidecar files in +an auxiliary directory owned by the MDT rather than values embedded in HFiles. +An MDT commit atomically publishes descriptors for already durable payloads. + +Queries are snapshot-safe. The default `complete` mode combines native index +results with a raw scan of source file slices not covered by a compatible +segment. An opt-in `fast` mode searches only covered data and reports that it +may omit matches. + +## Background + +Hudi indexes currently answer questions such as which files might contain a +record key or a value range. Full-text search has a different contract: analyze +free text into terms, locate matching documents, optionally verify positions, +and rank the best documents. Sending this workload to Elasticsearch or +OpenSearch is effective, but creates a second ingestion pipeline and a second +source of snapshot and retention truth. + +This proposal builds on: + +- [RFC-45](../rfc-45/rfc-45.md), which introduced asynchronous MDT indexing; +- [RFC-77](../rfc-77/rfc-77.md), which established dynamically named secondary + index partitions and index definitions; +- [RFC-102](../rfc-102/rfc-102.md), whose vector-search TVF provides a useful + SQL precedent; and +- RFC-109, the native vector-index proposal listed in the RFC catalog. Text and + vector search should eventually share native artifact packaging, storage + adapters, and top-k execution utilities, but their persistent formats remain + independent. + +### Design principles + +1. The Hudi timeline is the source of snapshot truth. +2. Index creation, visibility, rollback, and cleaning use MDT components. +3. Immutable payloads support object-store range reads and safe caching. +4. Ranking is independent of how the index is physically partitioned. +5. Freshness and incompleteness are explicit query properties. +6. The Java/native interface is small enough to replace without changing SQL. + +### Goals + +- Rank token, boolean, prefix, fuzzy, and phrase queries on one string column. +- Support copy-on-write (COW) and merge-on-read (MOR) tables. +- Build asynchronously and incrementally using the MDT indexer lifecycle. +- Guarantee snapshot-correct results in the default search mode. +- Provide an object-store-friendly native format with bounded memory usage. +- Keep the initial Rust implementation inside the Hudi repository. + +### Non-goals + +- Elasticsearch API, aggregation, highlighting, or percolator compatibility. +- Multi-column relevance models in the first format version. +- Updating posting lists in place. +- Replacing SQL predicate indexes or the record index. +- A general Rust rewrite of Hudi readers. + +### Alternatives considered + +**External Elasticsearch/OpenSearch.** This remains a valid integration, but it +requires change-data-capture coordination, separate retention, and explicit +mapping between external documents and a Hudi snapshot. + +**Embedding Tantivy.** Tantivy is mature and Rust-native. Its archive and +directory abstractions, however, become a second persistent compatibility +contract. A smaller Hudi-owned format gives the project control over source +file-slice identity, range-read layout, and MDT publication semantics. + +**Posting lists as MDT record values.** This would make MDT storage atomic, but +multi-gigabyte postings, merges, and random term reads fight the metadata +table's record-oriented HFile/MOR strengths. Small authoritative descriptors in +the MDT plus immutable sidecars preserve the lifecycle benefits without that +cost. + +## Implementation + +### Terminology + +| Term | Meaning | +| --- | --- | +| Logical index | A named SQL index and its immutable analyzer configuration. | +| MDT index partition | Dynamic `text_index_<name>` metadata partition containing authoritative control records. | +| Segment | An immutable set of documents and term postings built together. | +| Payload partition | A shard within a segment whose local document identifiers are `u32`. | +| Source slice | A Hudi base file and its ordered log files at a snapshot. | +| Coverage | Proof that a segment represents a particular source-slice fingerprint. | +| Raw tail | Eligible source slices not covered by compatible visible segments. | +| Document | One Hudi record, identified by record key and source information. | + +### SQL interface + +Creation follows Hudi's secondary-index syntax: + +```sql +CREATE INDEX article_body_fts +ON articles +USING text_index (body) +OPTIONS ( + 'base_tokenizer' = 'simple', + 'lower_case' = 'true', + 'language' = 'und', + 'with_position' = 'true', + 'posting_block_size' = '128' +); +``` + +The existing `HoodieIndexDefinition` is populated as follows: + +```json +{ + "indexName": "article_body_fts", + "indexType": "text_index", + "sourceFields": ["body"], + "indexFunction": "tokenize", + "indexOptions": { + "base_tokenizer": "simple", + "lower_case": "true", + "language": "und", + "with_position": "true", + "posting_block_size": "128" + } +} +``` + +Search is exposed as a Spark TVF rather than a boolean predicate because score +and top-k are part of its semantics: + +```sql +SELECT _hoodie_record_key, title, _score +FROM hudi_text_search( + table => 'articles', + column => 'body', + query => '{"match":{"query":"lakehouse indexing","operator":"and"}}', + k => 20, + options => map('search_mode', 'complete') +); +``` + +A phrase query is expressed as: + +```json +{"phrase":{"query":"metadata table","slop":1}} +``` + +The initial JSON query DSL contains `match`, `phrase`, `and`, `or`, and `not`. +Prefix and bounded edit-distance expansion are options on `match`. Unknown +fields and operators are rejected instead of ignored. Output includes the Hudi +record, `_score`, and `_hoodie_text_index` diagnostic metadata. Ties are ordered +by score descending, then partition path, record key, and file ID ascending. + +`search_mode` has three values: + +- `complete` (default): search compatible segments and raw-scan uncovered + source slices. Its result must equal a full raw search at the pinned snapshot. +- `fast`: search only compatible covered slices. The result metadata includes + coverage and sets `is_complete=false` when anything is skipped. +- `flat`: bypass the native index, useful as a correctness oracle and fallback. + +### Architecture + +```text + .hoodie/.index/index.json + | + v +Hudi timeline ---> text_index_<name> MDT partition ---> query coverage planner + | | | | + segment records coverage records | | + | | | + v v v +.hoodie/metadata/.aux/text-index/.../segment/ native splits raw splits + metadata.hfts part_*.tokens.hfts \ / + part_*.docs.hfts part_*.postings.hfts top-k merge + part_*.positions.hfts | + materialize rows +``` + +The SQL definition is stored in `.hoodie/.index/index.json`. A dynamic MDT +partition stores small control records. Immutable native payload files live +under an MDT-owned auxiliary namespace. Readers never infer visibility by +listing that namespace; they use descriptors visible in the pinned MDT +snapshot. + +### Metadata table integration + +Add `TEXT_INDEX` to `MetadataPartitionType` with the dynamic prefix +`text_index_`. `getPartitionPath(metaClient, indexName)` and index-definition +lookup follow the secondary and expression index conventions. Add a +`TextSearchIndexer` to `IndexerFactory`, implementing `BaseIndexer` lifecycle +operations. + +Add a tagged Avro metadata payload named `HoodieTextIndexInfo` to +`HoodieMetadata.avsc`. The record is deliberately descriptor-sized and has +four logical kinds: + +| Kind | Record key | Purpose | +| --- | --- | --- | +| `HEAD` | `head` | Format, analyzer fingerprint, publication instant, aggregate stats. | +| `SEGMENT` | `segment/<uuid>` | Payload paths, sizes, checksums, statistics, source-mask cardinality. | +| `COVERAGE` | `coverage/<encoded-partition>/<file-id>` | Source fingerprint and ordered candidate segments. | +| `TOMBSTONE` | `tombstone/<uuid>` | Segment retirement instant and deletion eligibility. | + +The record includes a schema version, index name, analyzer fingerprint, segment +UUID, payload format version, source fingerprints, source-ordinal mapping, +aggregate document and token counts, file descriptors, and optional tombstone +instant. Large term statistics, dictionaries, and postings are never placed in +the Avro record. Active-source masks are computed for the pinned snapshot rather +than persisted as a single current value. + +Payloads use this default path: + +```text +<table>/.hoodie/metadata/.aux/text-index/ + <escaped-index-name>/<segment-uuid>/... +``` + +The directory is below MDT ownership but outside normal MOR partition discovery. +A future external payload tier may be configured, but every path must be scoped +by table UUID and validated by readers. Only an MDT commit makes a segment +visible. Failed writers may leave unpublished files; the cleaner removes them +after a safety interval. + +### Source identity and snapshot coverage + +Each document stores this logical address: + +```text +HudiDocumentAddress { + source_ordinal: u32, + partition_path: bytes, + file_id: bytes, + record_key: bytes, + row_position_hint: optional u64 +} +``` + +Version 1 requires a stable Hudi record key. `file_id` lets materialization group +lookups, while `row_position_hint` is only an optimization and is used when the +source fingerprint matches exactly. + +A source fingerprint includes: + +- table UUID, partition path, and file ID; +- base instant and base-file identity (path, length, and checksum when known); +- ordered log-file identities (path, length, and latest block instant); +- writer schema identifier; and +- record-merger implementation and relevant options. + +At snapshot `S`, the coverage planner enumerates eligible file slices and +compares exact fingerprints. Exact matches activate the segment's source mask. +A changed base file or added MOR log makes that source uncovered until it is +rebuilt. A segment created from a later state is not used for an older snapshot. Review Comment: 🤖 Coverage requires an *exact* source-slice fingerprint match, so any new MOR log block, compaction, or clustering invalidates coverage and pushes that slice into the raw tail. For frequently-written or streaming tables, could you discuss how quickly the raw tail grows between catch-up builds, and what fraction of a typical query ends up being served by raw scan + per-query tail df analysis? It might be worth quantifying the freshness/latency envelope so users know when the index actually pays off. <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub> ########## rfc/rfc-110/rfc-110.md: ########## @@ -0,0 +1,639 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# RFC-110: Native Full-Text Search Index + +## Proposers + +- @danny0405 + +## Approvers + +- TBD + +## Status + +Issue: TBD + +> The RFC number is provisional until the community assigns an issue and +> accepts the proposal. RFC-109 is the highest numbered proposal in this +> checkout, so this draft uses RFC-110 to make repository review practical. + +## Abstract + +This RFC proposes a native, relevance-ranked full-text index for Apache Hudi. +It supports token and phrase search over string columns, exposes search through +a Spark table-valued function (TVF), and builds and maintains indexes through +the Hudi metadata table (MDT) indexing lifecycle. + +The search engine follows Lance's useful architectural choices without making +Lance a storage dependency: immutable segments, a compact term dictionary, +compressed posting lists, document-length statistics, BM25 ranking, positions, +and block-max WAND. Hudi owns the analyzer contract and on-disk format. The +format and hot search path are implemented in a Rust crate kept in the Hudi +repository and called through a narrow Java native boundary. + +The MDT remains authoritative for index definitions, visibility, coverage, +rollbacks, and cleaning. Large immutable posting payloads are sidecar files in +an auxiliary directory owned by the MDT rather than values embedded in HFiles. +An MDT commit atomically publishes descriptors for already durable payloads. + +Queries are snapshot-safe. The default `complete` mode combines native index +results with a raw scan of source file slices not covered by a compatible +segment. An opt-in `fast` mode searches only covered data and reports that it +may omit matches. + +## Background + +Hudi indexes currently answer questions such as which files might contain a +record key or a value range. Full-text search has a different contract: analyze +free text into terms, locate matching documents, optionally verify positions, +and rank the best documents. Sending this workload to Elasticsearch or +OpenSearch is effective, but creates a second ingestion pipeline and a second +source of snapshot and retention truth. + +This proposal builds on: + +- [RFC-45](../rfc-45/rfc-45.md), which introduced asynchronous MDT indexing; +- [RFC-77](../rfc-77/rfc-77.md), which established dynamically named secondary + index partitions and index definitions; +- [RFC-102](../rfc-102/rfc-102.md), whose vector-search TVF provides a useful + SQL precedent; and +- RFC-109, the native vector-index proposal listed in the RFC catalog. Text and + vector search should eventually share native artifact packaging, storage + adapters, and top-k execution utilities, but their persistent formats remain + independent. + +### Design principles + +1. The Hudi timeline is the source of snapshot truth. +2. Index creation, visibility, rollback, and cleaning use MDT components. +3. Immutable payloads support object-store range reads and safe caching. +4. Ranking is independent of how the index is physically partitioned. +5. Freshness and incompleteness are explicit query properties. +6. The Java/native interface is small enough to replace without changing SQL. + +### Goals + +- Rank token, boolean, prefix, fuzzy, and phrase queries on one string column. +- Support copy-on-write (COW) and merge-on-read (MOR) tables. +- Build asynchronously and incrementally using the MDT indexer lifecycle. +- Guarantee snapshot-correct results in the default search mode. +- Provide an object-store-friendly native format with bounded memory usage. +- Keep the initial Rust implementation inside the Hudi repository. + +### Non-goals + +- Elasticsearch API, aggregation, highlighting, or percolator compatibility. +- Multi-column relevance models in the first format version. +- Updating posting lists in place. +- Replacing SQL predicate indexes or the record index. +- A general Rust rewrite of Hudi readers. + +### Alternatives considered + +**External Elasticsearch/OpenSearch.** This remains a valid integration, but it +requires change-data-capture coordination, separate retention, and explicit +mapping between external documents and a Hudi snapshot. + +**Embedding Tantivy.** Tantivy is mature and Rust-native. Its archive and +directory abstractions, however, become a second persistent compatibility +contract. A smaller Hudi-owned format gives the project control over source +file-slice identity, range-read layout, and MDT publication semantics. + +**Posting lists as MDT record values.** This would make MDT storage atomic, but +multi-gigabyte postings, merges, and random term reads fight the metadata +table's record-oriented HFile/MOR strengths. Small authoritative descriptors in +the MDT plus immutable sidecars preserve the lifecycle benefits without that +cost. + +## Implementation + +### Terminology + +| Term | Meaning | +| --- | --- | +| Logical index | A named SQL index and its immutable analyzer configuration. | +| MDT index partition | Dynamic `text_index_<name>` metadata partition containing authoritative control records. | +| Segment | An immutable set of documents and term postings built together. | +| Payload partition | A shard within a segment whose local document identifiers are `u32`. | +| Source slice | A Hudi base file and its ordered log files at a snapshot. | +| Coverage | Proof that a segment represents a particular source-slice fingerprint. | +| Raw tail | Eligible source slices not covered by compatible visible segments. | +| Document | One Hudi record, identified by record key and source information. | + +### SQL interface + +Creation follows Hudi's secondary-index syntax: + +```sql +CREATE INDEX article_body_fts +ON articles +USING text_index (body) +OPTIONS ( + 'base_tokenizer' = 'simple', + 'lower_case' = 'true', + 'language' = 'und', + 'with_position' = 'true', + 'posting_block_size' = '128' +); +``` + +The existing `HoodieIndexDefinition` is populated as follows: + +```json +{ + "indexName": "article_body_fts", + "indexType": "text_index", + "sourceFields": ["body"], + "indexFunction": "tokenize", + "indexOptions": { + "base_tokenizer": "simple", + "lower_case": "true", + "language": "und", + "with_position": "true", + "posting_block_size": "128" + } +} +``` + +Search is exposed as a Spark TVF rather than a boolean predicate because score +and top-k are part of its semantics: + +```sql +SELECT _hoodie_record_key, title, _score Review Comment: 🤖 If an index is built with `with_position=false` (positions are optional per the format), what happens when a `phrase` query is issued against it? It might be worth stating whether the query is rejected, silently downgraded, or falls back to a raw scan, so the behavior is unambiguous at implementation time. <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub> ########## rfc/rfc-110/rfc-110.md: ########## @@ -0,0 +1,639 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# RFC-110: Native Full-Text Search Index + +## Proposers + +- @danny0405 + +## Approvers + +- TBD + +## Status + +Issue: TBD + +> The RFC number is provisional until the community assigns an issue and +> accepts the proposal. RFC-109 is the highest numbered proposal in this +> checkout, so this draft uses RFC-110 to make repository review practical. + +## Abstract + +This RFC proposes a native, relevance-ranked full-text index for Apache Hudi. +It supports token and phrase search over string columns, exposes search through +a Spark table-valued function (TVF), and builds and maintains indexes through +the Hudi metadata table (MDT) indexing lifecycle. + +The search engine follows Lance's useful architectural choices without making +Lance a storage dependency: immutable segments, a compact term dictionary, +compressed posting lists, document-length statistics, BM25 ranking, positions, +and block-max WAND. Hudi owns the analyzer contract and on-disk format. The +format and hot search path are implemented in a Rust crate kept in the Hudi +repository and called through a narrow Java native boundary. + +The MDT remains authoritative for index definitions, visibility, coverage, +rollbacks, and cleaning. Large immutable posting payloads are sidecar files in +an auxiliary directory owned by the MDT rather than values embedded in HFiles. +An MDT commit atomically publishes descriptors for already durable payloads. + +Queries are snapshot-safe. The default `complete` mode combines native index +results with a raw scan of source file slices not covered by a compatible +segment. An opt-in `fast` mode searches only covered data and reports that it +may omit matches. + +## Background + +Hudi indexes currently answer questions such as which files might contain a +record key or a value range. Full-text search has a different contract: analyze +free text into terms, locate matching documents, optionally verify positions, +and rank the best documents. Sending this workload to Elasticsearch or +OpenSearch is effective, but creates a second ingestion pipeline and a second +source of snapshot and retention truth. + +This proposal builds on: + +- [RFC-45](../rfc-45/rfc-45.md), which introduced asynchronous MDT indexing; +- [RFC-77](../rfc-77/rfc-77.md), which established dynamically named secondary + index partitions and index definitions; +- [RFC-102](../rfc-102/rfc-102.md), whose vector-search TVF provides a useful + SQL precedent; and +- RFC-109, the native vector-index proposal listed in the RFC catalog. Text and + vector search should eventually share native artifact packaging, storage + adapters, and top-k execution utilities, but their persistent formats remain + independent. + +### Design principles + +1. The Hudi timeline is the source of snapshot truth. +2. Index creation, visibility, rollback, and cleaning use MDT components. +3. Immutable payloads support object-store range reads and safe caching. +4. Ranking is independent of how the index is physically partitioned. +5. Freshness and incompleteness are explicit query properties. +6. The Java/native interface is small enough to replace without changing SQL. + +### Goals + +- Rank token, boolean, prefix, fuzzy, and phrase queries on one string column. +- Support copy-on-write (COW) and merge-on-read (MOR) tables. +- Build asynchronously and incrementally using the MDT indexer lifecycle. +- Guarantee snapshot-correct results in the default search mode. +- Provide an object-store-friendly native format with bounded memory usage. +- Keep the initial Rust implementation inside the Hudi repository. + +### Non-goals + +- Elasticsearch API, aggregation, highlighting, or percolator compatibility. +- Multi-column relevance models in the first format version. +- Updating posting lists in place. +- Replacing SQL predicate indexes or the record index. +- A general Rust rewrite of Hudi readers. + +### Alternatives considered + +**External Elasticsearch/OpenSearch.** This remains a valid integration, but it +requires change-data-capture coordination, separate retention, and explicit +mapping between external documents and a Hudi snapshot. + +**Embedding Tantivy.** Tantivy is mature and Rust-native. Its archive and +directory abstractions, however, become a second persistent compatibility +contract. A smaller Hudi-owned format gives the project control over source +file-slice identity, range-read layout, and MDT publication semantics. + +**Posting lists as MDT record values.** This would make MDT storage atomic, but +multi-gigabyte postings, merges, and random term reads fight the metadata +table's record-oriented HFile/MOR strengths. Small authoritative descriptors in +the MDT plus immutable sidecars preserve the lifecycle benefits without that +cost. + +## Implementation + +### Terminology + +| Term | Meaning | +| --- | --- | +| Logical index | A named SQL index and its immutable analyzer configuration. | +| MDT index partition | Dynamic `text_index_<name>` metadata partition containing authoritative control records. | +| Segment | An immutable set of documents and term postings built together. | +| Payload partition | A shard within a segment whose local document identifiers are `u32`. | +| Source slice | A Hudi base file and its ordered log files at a snapshot. | +| Coverage | Proof that a segment represents a particular source-slice fingerprint. | +| Raw tail | Eligible source slices not covered by compatible visible segments. | +| Document | One Hudi record, identified by record key and source information. | + +### SQL interface + +Creation follows Hudi's secondary-index syntax: + +```sql +CREATE INDEX article_body_fts +ON articles +USING text_index (body) +OPTIONS ( + 'base_tokenizer' = 'simple', + 'lower_case' = 'true', + 'language' = 'und', + 'with_position' = 'true', + 'posting_block_size' = '128' +); +``` + +The existing `HoodieIndexDefinition` is populated as follows: + +```json +{ + "indexName": "article_body_fts", + "indexType": "text_index", + "sourceFields": ["body"], + "indexFunction": "tokenize", + "indexOptions": { + "base_tokenizer": "simple", + "lower_case": "true", + "language": "und", + "with_position": "true", + "posting_block_size": "128" + } +} +``` + +Search is exposed as a Spark TVF rather than a boolean predicate because score +and top-k are part of its semantics: + +```sql +SELECT _hoodie_record_key, title, _score +FROM hudi_text_search( + table => 'articles', + column => 'body', + query => '{"match":{"query":"lakehouse indexing","operator":"and"}}', + k => 20, + options => map('search_mode', 'complete') +); +``` + +A phrase query is expressed as: + +```json +{"phrase":{"query":"metadata table","slop":1}} +``` + +The initial JSON query DSL contains `match`, `phrase`, `and`, `or`, and `not`. +Prefix and bounded edit-distance expansion are options on `match`. Unknown +fields and operators are rejected instead of ignored. Output includes the Hudi +record, `_score`, and `_hoodie_text_index` diagnostic metadata. Ties are ordered +by score descending, then partition path, record key, and file ID ascending. + +`search_mode` has three values: + +- `complete` (default): search compatible segments and raw-scan uncovered + source slices. Its result must equal a full raw search at the pinned snapshot. +- `fast`: search only compatible covered slices. The result metadata includes + coverage and sets `is_complete=false` when anything is skipped. +- `flat`: bypass the native index, useful as a correctness oracle and fallback. + +### Architecture + +```text + .hoodie/.index/index.json + | + v +Hudi timeline ---> text_index_<name> MDT partition ---> query coverage planner + | | | | + segment records coverage records | | + | | | + v v v +.hoodie/metadata/.aux/text-index/.../segment/ native splits raw splits + metadata.hfts part_*.tokens.hfts \ / + part_*.docs.hfts part_*.postings.hfts top-k merge + part_*.positions.hfts | + materialize rows +``` + +The SQL definition is stored in `.hoodie/.index/index.json`. A dynamic MDT +partition stores small control records. Immutable native payload files live +under an MDT-owned auxiliary namespace. Readers never infer visibility by +listing that namespace; they use descriptors visible in the pinned MDT +snapshot. + +### Metadata table integration + +Add `TEXT_INDEX` to `MetadataPartitionType` with the dynamic prefix +`text_index_`. `getPartitionPath(metaClient, indexName)` and index-definition +lookup follow the secondary and expression index conventions. Add a +`TextSearchIndexer` to `IndexerFactory`, implementing `BaseIndexer` lifecycle +operations. + +Add a tagged Avro metadata payload named `HoodieTextIndexInfo` to +`HoodieMetadata.avsc`. The record is deliberately descriptor-sized and has +four logical kinds: + +| Kind | Record key | Purpose | +| --- | --- | --- | +| `HEAD` | `head` | Format, analyzer fingerprint, publication instant, aggregate stats. | +| `SEGMENT` | `segment/<uuid>` | Payload paths, sizes, checksums, statistics, source-mask cardinality. | +| `COVERAGE` | `coverage/<encoded-partition>/<file-id>` | Source fingerprint and ordered candidate segments. | +| `TOMBSTONE` | `tombstone/<uuid>` | Segment retirement instant and deletion eligibility. | + +The record includes a schema version, index name, analyzer fingerprint, segment +UUID, payload format version, source fingerprints, source-ordinal mapping, +aggregate document and token counts, file descriptors, and optional tombstone +instant. Large term statistics, dictionaries, and postings are never placed in +the Avro record. Active-source masks are computed for the pinned snapshot rather +than persisted as a single current value. + +Payloads use this default path: Review Comment: 🤖 The COVERAGE record is keyed per (partition, file-id), i.e. roughly one control record per file group. For large tables with millions of file groups, how large does the `text_index_<name>` MDT partition get, and does the coverage planner read all coverage records on every query to resolve splits? Could you discuss the scaling behavior (and any pruning) of coverage resolution at that cardinality? <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub> ########## rfc/rfc-110/rfc-110.md: ########## @@ -0,0 +1,639 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# RFC-110: Native Full-Text Search Index + +## Proposers + +- @danny0405 + +## Approvers + +- TBD + +## Status + +Issue: TBD + +> The RFC number is provisional until the community assigns an issue and +> accepts the proposal. RFC-109 is the highest numbered proposal in this +> checkout, so this draft uses RFC-110 to make repository review practical. + +## Abstract + +This RFC proposes a native, relevance-ranked full-text index for Apache Hudi. +It supports token and phrase search over string columns, exposes search through +a Spark table-valued function (TVF), and builds and maintains indexes through +the Hudi metadata table (MDT) indexing lifecycle. + +The search engine follows Lance's useful architectural choices without making +Lance a storage dependency: immutable segments, a compact term dictionary, +compressed posting lists, document-length statistics, BM25 ranking, positions, +and block-max WAND. Hudi owns the analyzer contract and on-disk format. The +format and hot search path are implemented in a Rust crate kept in the Hudi +repository and called through a narrow Java native boundary. + +The MDT remains authoritative for index definitions, visibility, coverage, +rollbacks, and cleaning. Large immutable posting payloads are sidecar files in +an auxiliary directory owned by the MDT rather than values embedded in HFiles. +An MDT commit atomically publishes descriptors for already durable payloads. + +Queries are snapshot-safe. The default `complete` mode combines native index +results with a raw scan of source file slices not covered by a compatible +segment. An opt-in `fast` mode searches only covered data and reports that it +may omit matches. + +## Background + +Hudi indexes currently answer questions such as which files might contain a +record key or a value range. Full-text search has a different contract: analyze +free text into terms, locate matching documents, optionally verify positions, +and rank the best documents. Sending this workload to Elasticsearch or +OpenSearch is effective, but creates a second ingestion pipeline and a second +source of snapshot and retention truth. + +This proposal builds on: + +- [RFC-45](../rfc-45/rfc-45.md), which introduced asynchronous MDT indexing; +- [RFC-77](../rfc-77/rfc-77.md), which established dynamically named secondary + index partitions and index definitions; +- [RFC-102](../rfc-102/rfc-102.md), whose vector-search TVF provides a useful + SQL precedent; and +- RFC-109, the native vector-index proposal listed in the RFC catalog. Text and + vector search should eventually share native artifact packaging, storage + adapters, and top-k execution utilities, but their persistent formats remain + independent. + +### Design principles + +1. The Hudi timeline is the source of snapshot truth. +2. Index creation, visibility, rollback, and cleaning use MDT components. +3. Immutable payloads support object-store range reads and safe caching. +4. Ranking is independent of how the index is physically partitioned. +5. Freshness and incompleteness are explicit query properties. +6. The Java/native interface is small enough to replace without changing SQL. + +### Goals + +- Rank token, boolean, prefix, fuzzy, and phrase queries on one string column. +- Support copy-on-write (COW) and merge-on-read (MOR) tables. +- Build asynchronously and incrementally using the MDT indexer lifecycle. +- Guarantee snapshot-correct results in the default search mode. +- Provide an object-store-friendly native format with bounded memory usage. +- Keep the initial Rust implementation inside the Hudi repository. + +### Non-goals + +- Elasticsearch API, aggregation, highlighting, or percolator compatibility. +- Multi-column relevance models in the first format version. +- Updating posting lists in place. +- Replacing SQL predicate indexes or the record index. +- A general Rust rewrite of Hudi readers. + +### Alternatives considered + +**External Elasticsearch/OpenSearch.** This remains a valid integration, but it +requires change-data-capture coordination, separate retention, and explicit +mapping between external documents and a Hudi snapshot. + +**Embedding Tantivy.** Tantivy is mature and Rust-native. Its archive and +directory abstractions, however, become a second persistent compatibility +contract. A smaller Hudi-owned format gives the project control over source +file-slice identity, range-read layout, and MDT publication semantics. + +**Posting lists as MDT record values.** This would make MDT storage atomic, but +multi-gigabyte postings, merges, and random term reads fight the metadata +table's record-oriented HFile/MOR strengths. Small authoritative descriptors in +the MDT plus immutable sidecars preserve the lifecycle benefits without that +cost. + +## Implementation + +### Terminology + +| Term | Meaning | +| --- | --- | +| Logical index | A named SQL index and its immutable analyzer configuration. | +| MDT index partition | Dynamic `text_index_<name>` metadata partition containing authoritative control records. | +| Segment | An immutable set of documents and term postings built together. | +| Payload partition | A shard within a segment whose local document identifiers are `u32`. | +| Source slice | A Hudi base file and its ordered log files at a snapshot. | +| Coverage | Proof that a segment represents a particular source-slice fingerprint. | +| Raw tail | Eligible source slices not covered by compatible visible segments. | +| Document | One Hudi record, identified by record key and source information. | + +### SQL interface + +Creation follows Hudi's secondary-index syntax: + +```sql +CREATE INDEX article_body_fts +ON articles +USING text_index (body) +OPTIONS ( + 'base_tokenizer' = 'simple', + 'lower_case' = 'true', + 'language' = 'und', + 'with_position' = 'true', + 'posting_block_size' = '128' +); +``` + +The existing `HoodieIndexDefinition` is populated as follows: + +```json +{ + "indexName": "article_body_fts", + "indexType": "text_index", + "sourceFields": ["body"], + "indexFunction": "tokenize", + "indexOptions": { + "base_tokenizer": "simple", + "lower_case": "true", + "language": "und", + "with_position": "true", + "posting_block_size": "128" + } +} +``` + +Search is exposed as a Spark TVF rather than a boolean predicate because score +and top-k are part of its semantics: + +```sql +SELECT _hoodie_record_key, title, _score +FROM hudi_text_search( + table => 'articles', + column => 'body', + query => '{"match":{"query":"lakehouse indexing","operator":"and"}}', + k => 20, + options => map('search_mode', 'complete') +); +``` + +A phrase query is expressed as: + +```json +{"phrase":{"query":"metadata table","slop":1}} +``` + +The initial JSON query DSL contains `match`, `phrase`, `and`, `or`, and `not`. +Prefix and bounded edit-distance expansion are options on `match`. Unknown +fields and operators are rejected instead of ignored. Output includes the Hudi +record, `_score`, and `_hoodie_text_index` diagnostic metadata. Ties are ordered +by score descending, then partition path, record key, and file ID ascending. + +`search_mode` has three values: + +- `complete` (default): search compatible segments and raw-scan uncovered + source slices. Its result must equal a full raw search at the pinned snapshot. +- `fast`: search only compatible covered slices. The result metadata includes + coverage and sets `is_complete=false` when anything is skipped. +- `flat`: bypass the native index, useful as a correctness oracle and fallback. + +### Architecture + +```text + .hoodie/.index/index.json + | + v +Hudi timeline ---> text_index_<name> MDT partition ---> query coverage planner + | | | | + segment records coverage records | | + | | | + v v v +.hoodie/metadata/.aux/text-index/.../segment/ native splits raw splits + metadata.hfts part_*.tokens.hfts \ / + part_*.docs.hfts part_*.postings.hfts top-k merge + part_*.positions.hfts | + materialize rows +``` + +The SQL definition is stored in `.hoodie/.index/index.json`. A dynamic MDT +partition stores small control records. Immutable native payload files live +under an MDT-owned auxiliary namespace. Readers never infer visibility by +listing that namespace; they use descriptors visible in the pinned MDT +snapshot. + +### Metadata table integration + +Add `TEXT_INDEX` to `MetadataPartitionType` with the dynamic prefix +`text_index_`. `getPartitionPath(metaClient, indexName)` and index-definition +lookup follow the secondary and expression index conventions. Add a +`TextSearchIndexer` to `IndexerFactory`, implementing `BaseIndexer` lifecycle +operations. + +Add a tagged Avro metadata payload named `HoodieTextIndexInfo` to +`HoodieMetadata.avsc`. The record is deliberately descriptor-sized and has +four logical kinds: + +| Kind | Record key | Purpose | +| --- | --- | --- | +| `HEAD` | `head` | Format, analyzer fingerprint, publication instant, aggregate stats. | +| `SEGMENT` | `segment/<uuid>` | Payload paths, sizes, checksums, statistics, source-mask cardinality. | +| `COVERAGE` | `coverage/<encoded-partition>/<file-id>` | Source fingerprint and ordered candidate segments. | +| `TOMBSTONE` | `tombstone/<uuid>` | Segment retirement instant and deletion eligibility. | + +The record includes a schema version, index name, analyzer fingerprint, segment +UUID, payload format version, source fingerprints, source-ordinal mapping, +aggregate document and token counts, file descriptors, and optional tombstone +instant. Large term statistics, dictionaries, and postings are never placed in +the Avro record. Active-source masks are computed for the pinned snapshot rather +than persisted as a single current value. + +Payloads use this default path: + +```text +<table>/.hoodie/metadata/.aux/text-index/ + <escaped-index-name>/<segment-uuid>/... +``` + +The directory is below MDT ownership but outside normal MOR partition discovery. +A future external payload tier may be configured, but every path must be scoped +by table UUID and validated by readers. Only an MDT commit makes a segment +visible. Failed writers may leave unpublished files; the cleaner removes them +after a safety interval. + +### Source identity and snapshot coverage + +Each document stores this logical address: + +```text +HudiDocumentAddress { + source_ordinal: u32, + partition_path: bytes, + file_id: bytes, + record_key: bytes, + row_position_hint: optional u64 +} +``` + +Version 1 requires a stable Hudi record key. `file_id` lets materialization group +lookups, while `row_position_hint` is only an optimization and is used when the +source fingerprint matches exactly. + +A source fingerprint includes: + +- table UUID, partition path, and file ID; +- base instant and base-file identity (path, length, and checksum when known); +- ordered log-file identities (path, length, and latest block instant); +- writer schema identifier; and +- record-merger implementation and relevant options. + +At snapshot `S`, the coverage planner enumerates eligible file slices and +compares exact fingerprints. Exact matches activate the segment's source mask. +A changed base file or added MOR log makes that source uncovered until it is +rebuilt. A segment created from a later state is not used for an older snapshot. +This avoids attempting to delete or mutate old postings after compaction, +clustering, rollback, or MOR updates. + +### Native payload format + +All files begin with an eight-byte `HUDIFTS1` magic value followed by little- +endian format version, feature flags, variable-header length, and header +checksum. Independently checksummed blocks follow the header, and a footer +contains the block directory for range reads. Readers reject unknown required +feature bits and enforce configured allocation limits before reading lengths. + +Each segment contains: + +- `metadata.hfts`: analyzer fingerprint, source table, source descriptors, + segment statistics, payload partition descriptors, and checksums; +- `part_<n>.tokens.hfts`: a minimal finite-state transducer (FST) mapping + analyzed term bytes to term ordinals and posting metadata; +- `part_<n>.docs.hfts`: columnar source ordinal, record-key offsets and bytes, + document token count, and optional row-position hint; +- `part_<n>.postings.hfts`: document frequency, posting-block offsets, + delta-encoded local document IDs, term frequencies, and block-max metadata; + and +- `part_<n>.positions.hfts`: optional delta-encoded token positions and offsets. + +Payload partitions use local `u32` document IDs. A builder starts a new payload +partition before that space is exhausted. Posting blocks default to 128 +documents and use bit packing or variable-byte encoding, selected per block. +Each block records maximum term frequency and minimum document length; these +values provide a conservative BM25 upper bound for block-max WAND. Positions +are stored only when enabled by the immutable index definition. + +No Rust or collection serialization format is persisted directly. Every field +is defined by the Hudi format specification, so upgrading a dependency cannot +silently change files. + +### Rust implementation and Java boundary + +The first implementation is a top-level crate at `hudi-native-text-index`. The +module plan is: + +```text +format versioned headers, footers, checksums, block readers +analysis tokenizer and immutable analyzer pipeline +builder bounded-memory runs and segment construction +dictionary FST construction and lookup +postings compression, positions, and block-max metadata +query validated query AST and term expansion +scoring global-statistics BM25 +search conjunction, disjunction, phrase verification, top-k WAND +ffi opaque handles and panic-safe C ABI +``` + +The prototype committed with this RFC intentionally implements only document +addresses, segment statistics, BM25, and the fixed binary envelope. It makes +core formulas and compatibility failures executable while the wider design is +under review. + +The Java side exposes storage-neutral interfaces: + +```java +interface TextIndexInput { + long length(); + ByteBuffer readFully(long position, int length); +} + +interface TextIndexWriter { + SegmentDescriptor build(Iterator<TextDocument> documents, BuildOptions options); +} + +interface TextIndexReader extends AutoCloseable { + TermStatistics collectStatistics(QueryPlan query, SourceMask sources); + SearchResult search(QueryPlan query, GlobalStatistics stats, + SourceMask sources, int limit); +} +``` + +`TextIndexInput` is backed by `HoodieStorage`, preserving Hudi credentials, +range reads, retries, and metrics. Version 1 uses JNI, which supports Hudi's +Java 8/11 compatibility better than the Foreign Function and Memory API. +Native functions use opaque handles, explicit byte buffers, numeric error +codes, and a panic boundary; no C++ standard-library or Rust-owned collection +crosses the ABI. + +Native artifacts are published only for declared OS/architecture combinations +and selected through the existing Maven profile and classifier conventions. +When a native library is unavailable, `flat` search remains available. Hudi +must never silently interpret a persistent native segment using an incompatible +fallback reader. + +### Build and publication lifecycle + +A bootstrap build performs these steps: + +1. Pin a completed data-table instant and corresponding MDT snapshot. +2. Enumerate source file slices and construct their fingerprints. +3. Use Hudi's merged reader to emit stable record key, text, source ordinal, + and optional row-position hint. +4. Analyze documents and build bounded-memory sorted runs in Rust. +5. Merge runs into dictionaries, document tables, postings, and positions. +6. Write payload blocks to UUID-scoped temporary paths. +7. Finalize checksums, statistics, and source descriptors. +8. Move or copy payloads to their final immutable UUID paths when required by + the storage implementation. +9. Return `SEGMENT`, `COVERAGE`, and `HEAD` metadata records to the MDT writer. +10. Publish all descriptors and partition state in one MDT commit. Review Comment: 🤖 Publishing platform-specific compiled Rust artifacts (JNI) raises Apache release-policy questions — ASF releases are source releases, and shipping prebuilt native binaries per OS/arch has reproducibility and licensing implications. Could you add a section on how native artifacts get built, signed, and released within ASF conventions, and what the build story is for platforms you don't prebuild? @bvaradar this touches release/packaging policy and may need PMC input. <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
