danny0405 commented on code in PR #19613: URL: https://github.com/apache/hudi/pull/19613#discussion_r3789249757
########## rfc/rfc-full-text-search/rfc-full-text-search.md: ########## @@ -0,0 +1,891 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# RFC: Hudi Full-Text Search Index + +## Proposers + +- @danny0405 + +## Approvers + +- TBD + +## Status + +Issue: TBD + +> This proposal intentionally does not claim an RFC number. The number and +> catalog entry will be reserved through the separate RFC-number process. + +## Abstract + +This RFC proposes a full-text index for Apache Hudi. It supports token, phrase, +and multi-column search over string columns, exposes standard predicates through +Spark SQL, and offers a Lance-style direct table API through Hudi RS Python. +Indexes are built and maintained through the Hudi metadata table (MDT) +indexing lifecycle. + +The search engine follows Lance's useful architectural choices without making +Lance a storage dependency: immutable segments, a compact term dictionary, +compressed posting lists, document-length statistics, BM25 ranking, positions, +and block-max WAND. Hudi owns the analyzer contract and on-disk format. Spark +integration remains in the main Hudi repository; Rust and Python implementation +work belongs in Hudi RS and consumes the same versioned format. + +The MDT remains authoritative for index definitions, visibility, coverage, +rollbacks, and cleaning. Large immutable posting payloads are sidecar files in +an auxiliary directory owned by the MDT rather than values embedded in HFiles. +An MDT commit atomically publishes descriptors for already durable payloads. + +Queries are snapshot-safe. SQL predicates always combine index results with a +raw scan of source file slices not covered by a compatible segment, so using an +index never changes SQL results. The direct search API also defaults to complete +results and may offer an explicitly incomplete low-latency mode. + +## Background + +Hudi indexes currently answer questions such as which files might contain a +record key or a value range. Full-text search has a different contract: analyze +free text into terms, locate matching documents, optionally verify positions, +and, for direct search APIs, rank the best documents. Sending this workload to +Elasticsearch or OpenSearch is effective, but creates a second ingestion +pipeline and a second source of snapshot and retention truth. + +This proposal builds on: + +- [RFC-45](../rfc-45/rfc-45.md), which introduced asynchronous MDT indexing; +- [RFC-77](../rfc-77/rfc-77.md), which established dynamically named secondary + index partitions and index definitions; +- the standard Spark SQL predicate model, which keeps index acceleration + transparent to relational queries; and +- RFC-109, the native vector-index proposal listed in the RFC catalog. Text and + vector search may share Hudi RS storage adapters and top-k utilities, but + their persistent formats remain independent. + +### Design principles + +1. The Hudi timeline is the source of snapshot truth. +2. Index creation, visibility, rollback, and cleaning use MDT components. +3. Immutable payloads support object-store range reads and safe caching. Review Comment: Addressed. The ambiguous term payload has been removed. The RFC now defines immutable index data files: a published file is never edited in place; updates write replacement files and atomically switch the MDT references. ########## rfc/rfc-full-text-search/rfc-full-text-search.md: ########## @@ -0,0 +1,891 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# RFC: Hudi Full-Text Search Index + +## Proposers + +- @danny0405 + +## Approvers + +- TBD + +## Status + +Issue: TBD + +> This proposal intentionally does not claim an RFC number. The number and +> catalog entry will be reserved through the separate RFC-number process. + +## Abstract + +This RFC proposes a full-text index for Apache Hudi. It supports token, phrase, +and multi-column search over string columns, exposes standard predicates through +Spark SQL, and offers a Lance-style direct table API through Hudi RS Python. +Indexes are built and maintained through the Hudi metadata table (MDT) +indexing lifecycle. + +The search engine follows Lance's useful architectural choices without making +Lance a storage dependency: immutable segments, a compact term dictionary, +compressed posting lists, document-length statistics, BM25 ranking, positions, +and block-max WAND. Hudi owns the analyzer contract and on-disk format. Spark +integration remains in the main Hudi repository; Rust and Python implementation +work belongs in Hudi RS and consumes the same versioned format. + +The MDT remains authoritative for index definitions, visibility, coverage, +rollbacks, and cleaning. Large immutable posting payloads are sidecar files in +an auxiliary directory owned by the MDT rather than values embedded in HFiles. +An MDT commit atomically publishes descriptors for already durable payloads. + +Queries are snapshot-safe. SQL predicates always combine index results with a +raw scan of source file slices not covered by a compatible segment, so using an +index never changes SQL results. The direct search API also defaults to complete +results and may offer an explicitly incomplete low-latency mode. + +## Background + +Hudi indexes currently answer questions such as which files might contain a +record key or a value range. Full-text search has a different contract: analyze +free text into terms, locate matching documents, optionally verify positions, +and, for direct search APIs, rank the best documents. Sending this workload to +Elasticsearch or OpenSearch is effective, but creates a second ingestion +pipeline and a second source of snapshot and retention truth. + +This proposal builds on: + +- [RFC-45](../rfc-45/rfc-45.md), which introduced asynchronous MDT indexing; +- [RFC-77](../rfc-77/rfc-77.md), which established dynamically named secondary + index partitions and index definitions; +- the standard Spark SQL predicate model, which keeps index acceleration + transparent to relational queries; and +- RFC-109, the native vector-index proposal listed in the RFC catalog. Text and + vector search may share Hudi RS storage adapters and top-k utilities, but + their persistent formats remain independent. + +### Design principles + +1. The Hudi timeline is the source of snapshot truth. +2. Index creation, visibility, rollback, and cleaning use MDT components. +3. Immutable payloads support object-store range reads and safe caching. +4. Index use never changes the result of a Spark SQL predicate. +5. Ranking is independent of how the index is physically partitioned. +6. JVM and Hudi RS clients share query semantics and format versions. + +### Goals + +- Match token, boolean, prefix, fuzzy, phrase, and multi-column queries. +- Support copy-on-write (COW) and merge-on-read (MOR) tables. +- Build asynchronously and incrementally using the MDT indexer lifecycle. +- Guarantee snapshot-correct results for SQL and the default direct API mode. Review Comment: Addressed. The SQL interface now explicitly says SQL means Spark SQL backed by Java in the main Hudi repository, while the direct API means Hudi RS Python backed by Rust. The two implementations share persisted contracts and fixtures, not a runtime dependency. ########## rfc/rfc-full-text-search/rfc-full-text-search.md: ########## @@ -0,0 +1,891 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# RFC: Hudi Full-Text Search Index + +## Proposers + +- @danny0405 + +## Approvers + +- TBD + +## Status + +Issue: TBD + +> This proposal intentionally does not claim an RFC number. The number and +> catalog entry will be reserved through the separate RFC-number process. + +## Abstract + +This RFC proposes a full-text index for Apache Hudi. It supports token, phrase, +and multi-column search over string columns, exposes standard predicates through +Spark SQL, and offers a Lance-style direct table API through Hudi RS Python. +Indexes are built and maintained through the Hudi metadata table (MDT) +indexing lifecycle. + +The search engine follows Lance's useful architectural choices without making +Lance a storage dependency: immutable segments, a compact term dictionary, +compressed posting lists, document-length statistics, BM25 ranking, positions, +and block-max WAND. Hudi owns the analyzer contract and on-disk format. Spark +integration remains in the main Hudi repository; Rust and Python implementation +work belongs in Hudi RS and consumes the same versioned format. + +The MDT remains authoritative for index definitions, visibility, coverage, +rollbacks, and cleaning. Large immutable posting payloads are sidecar files in +an auxiliary directory owned by the MDT rather than values embedded in HFiles. +An MDT commit atomically publishes descriptors for already durable payloads. + +Queries are snapshot-safe. SQL predicates always combine index results with a +raw scan of source file slices not covered by a compatible segment, so using an +index never changes SQL results. The direct search API also defaults to complete +results and may offer an explicitly incomplete low-latency mode. + +## Background + +Hudi indexes currently answer questions such as which files might contain a +record key or a value range. Full-text search has a different contract: analyze +free text into terms, locate matching documents, optionally verify positions, +and, for direct search APIs, rank the best documents. Sending this workload to +Elasticsearch or OpenSearch is effective, but creates a second ingestion +pipeline and a second source of snapshot and retention truth. + +This proposal builds on: + +- [RFC-45](../rfc-45/rfc-45.md), which introduced asynchronous MDT indexing; +- [RFC-77](../rfc-77/rfc-77.md), which established dynamically named secondary + index partitions and index definitions; +- the standard Spark SQL predicate model, which keeps index acceleration + transparent to relational queries; and +- RFC-109, the native vector-index proposal listed in the RFC catalog. Text and + vector search may share Hudi RS storage adapters and top-k utilities, but + their persistent formats remain independent. + +### Design principles + +1. The Hudi timeline is the source of snapshot truth. +2. Index creation, visibility, rollback, and cleaning use MDT components. +3. Immutable payloads support object-store range reads and safe caching. +4. Index use never changes the result of a Spark SQL predicate. +5. Ranking is independent of how the index is physically partitioned. +6. JVM and Hudi RS clients share query semantics and format versions. + +### Goals + +- Match token, boolean, prefix, fuzzy, phrase, and multi-column queries. +- Support copy-on-write (COW) and merge-on-read (MOR) tables. +- Build asynchronously and incrementally using the MDT indexer lifecycle. +- Guarantee snapshot-correct results for SQL and the default direct API mode. +- Provide an object-store-friendly text-index format with bounded memory usage. Review Comment: Addressed. The Goals section now explains that object storage provides persistence but not a memory bound: a frequent term can reference millions of records. Block directories, range reads, bounded caches, streaming decoders, spillable builders, and query expansion limits keep task memory independent of the whole segment size. ########## rfc/rfc-full-text-search/rfc-full-text-search.md: ########## @@ -0,0 +1,891 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# RFC: Hudi Full-Text Search Index + +## Proposers + +- @danny0405 + +## Approvers + +- TBD + +## Status + +Issue: TBD + +> This proposal intentionally does not claim an RFC number. The number and +> catalog entry will be reserved through the separate RFC-number process. + +## Abstract + +This RFC proposes a full-text index for Apache Hudi. It supports token, phrase, +and multi-column search over string columns, exposes standard predicates through +Spark SQL, and offers a Lance-style direct table API through Hudi RS Python. +Indexes are built and maintained through the Hudi metadata table (MDT) +indexing lifecycle. + +The search engine follows Lance's useful architectural choices without making +Lance a storage dependency: immutable segments, a compact term dictionary, +compressed posting lists, document-length statistics, BM25 ranking, positions, +and block-max WAND. Hudi owns the analyzer contract and on-disk format. Spark +integration remains in the main Hudi repository; Rust and Python implementation +work belongs in Hudi RS and consumes the same versioned format. + +The MDT remains authoritative for index definitions, visibility, coverage, +rollbacks, and cleaning. Large immutable posting payloads are sidecar files in +an auxiliary directory owned by the MDT rather than values embedded in HFiles. +An MDT commit atomically publishes descriptors for already durable payloads. + +Queries are snapshot-safe. SQL predicates always combine index results with a +raw scan of source file slices not covered by a compatible segment, so using an +index never changes SQL results. The direct search API also defaults to complete +results and may offer an explicitly incomplete low-latency mode. + +## Background + +Hudi indexes currently answer questions such as which files might contain a +record key or a value range. Full-text search has a different contract: analyze +free text into terms, locate matching documents, optionally verify positions, +and, for direct search APIs, rank the best documents. Sending this workload to +Elasticsearch or OpenSearch is effective, but creates a second ingestion +pipeline and a second source of snapshot and retention truth. + +This proposal builds on: + +- [RFC-45](../rfc-45/rfc-45.md), which introduced asynchronous MDT indexing; +- [RFC-77](../rfc-77/rfc-77.md), which established dynamically named secondary + index partitions and index definitions; +- the standard Spark SQL predicate model, which keeps index acceleration + transparent to relational queries; and +- RFC-109, the native vector-index proposal listed in the RFC catalog. Text and + vector search may share Hudi RS storage adapters and top-k utilities, but + their persistent formats remain independent. + +### Design principles + +1. The Hudi timeline is the source of snapshot truth. +2. Index creation, visibility, rollback, and cleaning use MDT components. +3. Immutable payloads support object-store range reads and safe caching. +4. Index use never changes the result of a Spark SQL predicate. +5. Ranking is independent of how the index is physically partitioned. +6. JVM and Hudi RS clients share query semantics and format versions. + +### Goals + +- Match token, boolean, prefix, fuzzy, phrase, and multi-column queries. +- Support copy-on-write (COW) and merge-on-read (MOR) tables. +- Build asynchronously and incrementally using the MDT indexer lifecycle. +- Guarantee snapshot-correct results for SQL and the default direct API mode. +- Provide an object-store-friendly text-index format with bounded memory usage. +- Provide a direct Hudi RS Python API alongside Spark SQL. + +### Non-goals + +- Elasticsearch API, aggregation, highlighting, or percolator compatibility. +- Highlighting and custom relevance models in the first format version. +- Updating posting lists in place. +- Replacing SQL predicate indexes or the record index. +- Adding Rust code or native build integration to the main Hudi repository. + +### Alternatives considered + +**External Elasticsearch/OpenSearch.** This remains a valid integration, but it Review Comment: Addressed. I removed the speculative alternatives/integration discussion and added the concrete Motivation and use cases section explaining why snapshot-aware search belongs with lakehouse storage. Hybrid ranking is now explicitly deferred to a separate RFC with no runtime dependency on RFC-109. ########## rfc/rfc-full-text-search/rfc-full-text-search.md: ########## @@ -0,0 +1,891 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# RFC: Hudi Full-Text Search Index + +## Proposers + +- @danny0405 + +## Approvers + +- TBD + +## Status + +Issue: TBD + +> This proposal intentionally does not claim an RFC number. The number and +> catalog entry will be reserved through the separate RFC-number process. + +## Abstract + +This RFC proposes a full-text index for Apache Hudi. It supports token, phrase, +and multi-column search over string columns, exposes standard predicates through +Spark SQL, and offers a Lance-style direct table API through Hudi RS Python. +Indexes are built and maintained through the Hudi metadata table (MDT) +indexing lifecycle. + +The search engine follows Lance's useful architectural choices without making +Lance a storage dependency: immutable segments, a compact term dictionary, +compressed posting lists, document-length statistics, BM25 ranking, positions, +and block-max WAND. Hudi owns the analyzer contract and on-disk format. Spark +integration remains in the main Hudi repository; Rust and Python implementation +work belongs in Hudi RS and consumes the same versioned format. + +The MDT remains authoritative for index definitions, visibility, coverage, +rollbacks, and cleaning. Large immutable posting payloads are sidecar files in +an auxiliary directory owned by the MDT rather than values embedded in HFiles. +An MDT commit atomically publishes descriptors for already durable payloads. + +Queries are snapshot-safe. SQL predicates always combine index results with a +raw scan of source file slices not covered by a compatible segment, so using an +index never changes SQL results. The direct search API also defaults to complete +results and may offer an explicitly incomplete low-latency mode. + +## Background + +Hudi indexes currently answer questions such as which files might contain a +record key or a value range. Full-text search has a different contract: analyze +free text into terms, locate matching documents, optionally verify positions, +and, for direct search APIs, rank the best documents. Sending this workload to +Elasticsearch or OpenSearch is effective, but creates a second ingestion +pipeline and a second source of snapshot and retention truth. + +This proposal builds on: + +- [RFC-45](../rfc-45/rfc-45.md), which introduced asynchronous MDT indexing; +- [RFC-77](../rfc-77/rfc-77.md), which established dynamically named secondary + index partitions and index definitions; +- the standard Spark SQL predicate model, which keeps index acceleration + transparent to relational queries; and +- RFC-109, the native vector-index proposal listed in the RFC catalog. Text and + vector search may share Hudi RS storage adapters and top-k utilities, but + their persistent formats remain independent. + +### Design principles + +1. The Hudi timeline is the source of snapshot truth. +2. Index creation, visibility, rollback, and cleaning use MDT components. +3. Immutable payloads support object-store range reads and safe caching. +4. Index use never changes the result of a Spark SQL predicate. +5. Ranking is independent of how the index is physically partitioned. +6. JVM and Hudi RS clients share query semantics and format versions. + +### Goals + +- Match token, boolean, prefix, fuzzy, phrase, and multi-column queries. +- Support copy-on-write (COW) and merge-on-read (MOR) tables. +- Build asynchronously and incrementally using the MDT indexer lifecycle. +- Guarantee snapshot-correct results for SQL and the default direct API mode. +- Provide an object-store-friendly text-index format with bounded memory usage. +- Provide a direct Hudi RS Python API alongside Spark SQL. + +### Non-goals + +- Elasticsearch API, aggregation, highlighting, or percolator compatibility. +- Highlighting and custom relevance models in the first format version. +- Updating posting lists in place. +- Replacing SQL predicate indexes or the record index. +- Adding Rust code or native build integration to the main Hudi repository. + +### Alternatives considered + +**External Elasticsearch/OpenSearch.** This remains a valid integration, but it +requires change-data-capture coordination, separate retention, and explicit +mapping between external documents and a Hudi snapshot. + +**Embedding Tantivy.** Tantivy is mature and Rust-native. Its archive and +directory abstractions, however, become a second persistent compatibility +contract. A smaller Hudi-owned format gives the project control over source +file-slice identity, range-read layout, and MDT publication semantics. + +**Posting lists as MDT record values.** This would make MDT storage atomic, but Review Comment: Addressed. Background and the terminology table now define a posting list as the ordered local record ordinals containing one term, with term frequencies and optional positions. ########## rfc/rfc-full-text-search/rfc-full-text-search.md: ########## @@ -0,0 +1,891 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# RFC: Hudi Full-Text Search Index + +## Proposers + +- @danny0405 + +## Approvers + +- TBD + +## Status + +Issue: TBD + +> This proposal intentionally does not claim an RFC number. The number and +> catalog entry will be reserved through the separate RFC-number process. + +## Abstract + +This RFC proposes a full-text index for Apache Hudi. It supports token, phrase, +and multi-column search over string columns, exposes standard predicates through +Spark SQL, and offers a Lance-style direct table API through Hudi RS Python. +Indexes are built and maintained through the Hudi metadata table (MDT) +indexing lifecycle. + +The search engine follows Lance's useful architectural choices without making +Lance a storage dependency: immutable segments, a compact term dictionary, +compressed posting lists, document-length statistics, BM25 ranking, positions, +and block-max WAND. Hudi owns the analyzer contract and on-disk format. Spark +integration remains in the main Hudi repository; Rust and Python implementation +work belongs in Hudi RS and consumes the same versioned format. + +The MDT remains authoritative for index definitions, visibility, coverage, +rollbacks, and cleaning. Large immutable posting payloads are sidecar files in +an auxiliary directory owned by the MDT rather than values embedded in HFiles. +An MDT commit atomically publishes descriptors for already durable payloads. + +Queries are snapshot-safe. SQL predicates always combine index results with a +raw scan of source file slices not covered by a compatible segment, so using an +index never changes SQL results. The direct search API also defaults to complete +results and may offer an explicitly incomplete low-latency mode. + +## Background + +Hudi indexes currently answer questions such as which files might contain a +record key or a value range. Full-text search has a different contract: analyze +free text into terms, locate matching documents, optionally verify positions, +and, for direct search APIs, rank the best documents. Sending this workload to +Elasticsearch or OpenSearch is effective, but creates a second ingestion +pipeline and a second source of snapshot and retention truth. + +This proposal builds on: + +- [RFC-45](../rfc-45/rfc-45.md), which introduced asynchronous MDT indexing; +- [RFC-77](../rfc-77/rfc-77.md), which established dynamically named secondary + index partitions and index definitions; +- the standard Spark SQL predicate model, which keeps index acceleration + transparent to relational queries; and +- RFC-109, the native vector-index proposal listed in the RFC catalog. Text and + vector search may share Hudi RS storage adapters and top-k utilities, but + their persistent formats remain independent. + +### Design principles + +1. The Hudi timeline is the source of snapshot truth. +2. Index creation, visibility, rollback, and cleaning use MDT components. +3. Immutable payloads support object-store range reads and safe caching. +4. Index use never changes the result of a Spark SQL predicate. +5. Ranking is independent of how the index is physically partitioned. +6. JVM and Hudi RS clients share query semantics and format versions. + +### Goals + +- Match token, boolean, prefix, fuzzy, phrase, and multi-column queries. +- Support copy-on-write (COW) and merge-on-read (MOR) tables. +- Build asynchronously and incrementally using the MDT indexer lifecycle. +- Guarantee snapshot-correct results for SQL and the default direct API mode. +- Provide an object-store-friendly text-index format with bounded memory usage. +- Provide a direct Hudi RS Python API alongside Spark SQL. + +### Non-goals + +- Elasticsearch API, aggregation, highlighting, or percolator compatibility. +- Highlighting and custom relevance models in the first format version. +- Updating posting lists in place. +- Replacing SQL predicate indexes or the record index. +- Adding Rust code or native build integration to the main Hudi repository. + +### Alternatives considered + +**External Elasticsearch/OpenSearch.** This remains a valid integration, but it +requires change-data-capture coordination, separate retention, and explicit +mapping between external documents and a Hudi snapshot. + +**Embedding Tantivy.** Tantivy is mature and Rust-native. Its archive and +directory abstractions, however, become a second persistent compatibility +contract. A smaller Hudi-owned format gives the project control over source +file-slice identity, range-read layout, and MDT publication semantics. + +**Posting lists as MDT record values.** This would make MDT storage atomic, but +multi-gigabyte postings, merges, and random term reads fight the metadata +table's record-oriented HFile/MOR strengths. Small authoritative descriptors in +the MDT plus immutable sidecars preserve the lifecycle benefits without that +cost. + +## Implementation + +### Terminology + +| Term | Meaning | Review Comment: Addressed throughout the RFC. The primary vocabulary is now data table file group, data table file slice, Hudi record key, MDT partition, file-slice identity, coverage record, and fallback scan set. Search-specific terms are retained only where defined. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
