danny0405 commented on code in PR #19613: URL: https://github.com/apache/hudi/pull/19613#discussion_r3789249507
########## rfc/rfc-full-text-search/rfc-full-text-search.md: ########## @@ -0,0 +1,639 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# RFC-110: Native Full-Text Search Index + +## Proposers + +- @danny0405 + +## Approvers + +- TBD + +## Status + +Issue: TBD + +> The RFC number is provisional until the community assigns an issue and +> accepts the proposal. RFC-109 is the highest numbered proposal in this +> checkout, so this draft uses RFC-110 to make repository review practical. + +## Abstract + +This RFC proposes a native, relevance-ranked full-text index for Apache Hudi. +It supports token and phrase search over string columns, exposes search through +a Spark table-valued function (TVF), and builds and maintains indexes through +the Hudi metadata table (MDT) indexing lifecycle. + +The search engine follows Lance's useful architectural choices without making +Lance a storage dependency: immutable segments, a compact term dictionary, +compressed posting lists, document-length statistics, BM25 ranking, positions, +and block-max WAND. Hudi owns the analyzer contract and on-disk format. The +format and hot search path are implemented in a Rust crate kept in the Hudi +repository and called through a narrow Java native boundary. + +The MDT remains authoritative for index definitions, visibility, coverage, +rollbacks, and cleaning. Large immutable posting payloads are sidecar files in +an auxiliary directory owned by the MDT rather than values embedded in HFiles. +An MDT commit atomically publishes descriptors for already durable payloads. + +Queries are snapshot-safe. The default `complete` mode combines native index +results with a raw scan of source file slices not covered by a compatible +segment. An opt-in `fast` mode searches only covered data and reports that it +may omit matches. + +## Background + +Hudi indexes currently answer questions such as which files might contain a +record key or a value range. Full-text search has a different contract: analyze +free text into terms, locate matching documents, optionally verify positions, +and rank the best documents. Sending this workload to Elasticsearch or +OpenSearch is effective, but creates a second ingestion pipeline and a second +source of snapshot and retention truth. + +This proposal builds on: + +- [RFC-45](../rfc-45/rfc-45.md), which introduced asynchronous MDT indexing; +- [RFC-77](../rfc-77/rfc-77.md), which established dynamically named secondary + index partitions and index definitions; +- [RFC-102](../rfc-102/rfc-102.md), whose vector-search TVF provides a useful + SQL precedent; and +- RFC-109, the native vector-index proposal listed in the RFC catalog. Text and + vector search should eventually share native artifact packaging, storage + adapters, and top-k execution utilities, but their persistent formats remain + independent. + +### Design principles Review Comment: Addressed. The new Motivation and use cases section covers logs and observability, catalog and support text, security and eDiscovery, a concrete GDPR deletion-lag failure, time-travel search, and future hybrid text/vector retrieval without coupling RFC-110 to RFC-109. The SQL and Python examples now use application logs and support tickets. ########## rfc/README.md: ########## @@ -145,3 +145,4 @@ The list of all RFCs can be found here. | 107 | Dynamic Partitioned Cache for Flink upsert | :hammer_and_wrench: `IN PROGRESS` | | 108 | [Multi-dataset incremental reads in Hudi Streamer](./rfc-108/rfc-108.md) | :eyes: `UNDER REVIEW` | | 109 | Hudi Native Vector Index | :eyes: `UNDER REVIEW` | +| 110 | [Native Full-Text Search Index](./rfc-110/rfc-110.md) | :eyes: `UNDER REVIEW` | Review Comment: Addressed through the project process. RFC-110 was reserved separately by #19614 and that PR is merged. This PR now uses RFC-110 in the document path, heading, catalog link, and PR title, and links the reservation PR from the Status section. ########## rfc/rfc-full-text-search/rfc-full-text-search.md: ########## @@ -0,0 +1,891 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# RFC: Hudi Full-Text Search Index + +## Proposers + +- @danny0405 + +## Approvers + +- TBD Review Comment: Thank you. I added @vinothchandar under the RFC Reviewers section. ########## rfc/rfc-full-text-search/rfc-full-text-search.md: ########## @@ -0,0 +1,891 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# RFC: Hudi Full-Text Search Index + +## Proposers + +- @danny0405 + +## Approvers + +- TBD + +## Status + +Issue: TBD + +> This proposal intentionally does not claim an RFC number. The number and +> catalog entry will be reserved through the separate RFC-number process. + +## Abstract + +This RFC proposes a full-text index for Apache Hudi. It supports token, phrase, +and multi-column search over string columns, exposes standard predicates through +Spark SQL, and offers a Lance-style direct table API through Hudi RS Python. Review Comment: Addressed. The RFC now explains that the design draws from Lucene, Elasticsearch/OpenSearch, Tantivy, Paimon, and Lance, and links their primary specifications or documentation. Hybrid text/vector retrieval is identified as a future user benefit but remains outside this RFC. ########## rfc/rfc-full-text-search/rfc-full-text-search.md: ########## @@ -0,0 +1,891 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# RFC: Hudi Full-Text Search Index + +## Proposers + +- @danny0405 + +## Approvers + +- TBD + +## Status + +Issue: TBD + +> This proposal intentionally does not claim an RFC number. The number and +> catalog entry will be reserved through the separate RFC-number process. + +## Abstract + +This RFC proposes a full-text index for Apache Hudi. It supports token, phrase, +and multi-column search over string columns, exposes standard predicates through +Spark SQL, and offers a Lance-style direct table API through Hudi RS Python. +Indexes are built and maintained through the Hudi metadata table (MDT) +indexing lifecycle. + +The search engine follows Lance's useful architectural choices without making +Lance a storage dependency: immutable segments, a compact term dictionary, +compressed posting lists, document-length statistics, BM25 ranking, positions, +and block-max WAND. Hudi owns the analyzer contract and on-disk format. Spark +integration remains in the main Hudi repository; Rust and Python implementation +work belongs in Hudi RS and consumes the same versioned format. + +The MDT remains authoritative for index definitions, visibility, coverage, +rollbacks, and cleaning. Large immutable posting payloads are sidecar files in Review Comment: Addressed. Background now defines a posting and posting list in terms of matching Hudi records. I removed the term sidecar and consistently call the external objects immutable index data files, with MDT records controlling their visibility and lifecycle. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
