chrevanthreddy opened a new pull request, #19802: URL: https://github.com/apache/hudi/pull/19802
### Describe the issue this Pull Request addresses Implements the vector-index design proposed in RFC-109 (#19309), providing native approximate nearest-neighbor search over Hudi vector columns while retaining the data table as the authoritative source for exact re-ranking. The implementation uses the Metadata Table for IVF routing, residual RaBitQ postings, freshness arbitration, and incremental maintenance. It also adds positional/key-based exact fetch across COW and MOR file slices. ### Summary and Changelog - Add vector index definitions and SQL/API option handling. - Add IVF training/routing and residual 4-bit RaBitQ encoding/scoring. - Store centroids, quantizer state, manifests, and posting blocks in a vector MDT partition. - Add approximate candidate generation and optional exact re-ranking. - Integrate RLI freshness arbitration and stale-posting suppression. - Add COW/MOR exact fetch, including merged-slice handling for log-resident records. - Add incremental index maintenance for inserts, updates, and deletes. - Treat stored row positions as hints and require record-key verification before exact scoring. - Add unit and bounded Spark integration coverage for bootstrap, query, update, delete, and MOR behavior. No code was copied. ### Impact Adds an opt-in vector-index feature. Existing tables and non-vector readers/writers are unchanged unless a vector index is explicitly created. The patch adds MDT vector partitions and Spark SQL planning/execution paths. ### Risk Level high — this is a substantial new indexing and query path. The PR is intentionally opened as a draft while the final acceptance battery completes. Completed evidence: - BIGANN 1B COW ingest: 1B rows / 280 files. - 16,384-cluster, 4-bit residual RaBitQ bootstrap completed. - Approximate recall@10: 0.822 / 0.857 / 0.871 at nprobe 16 / 32 / 64. - Exact nprobe=32/refine=50 recall@10: 0.960. - Bounded COW/MOR update and exact-vs-brute-force tests pass. - `TestVectorIndexOptions`: 10 passed, including mandatory key verification. Still running or required before ready-for-review: - Corrected 10M COW/MOR lifecycle battery with physical 512-file-group assertion. - Delete/replay/stale-posting and clustering/compaction relocation gates. - Corrected 1B rebuild if the 512-group proof passes. - Bounded 1B post-bootstrap mutation test. - Independent billion-key RLI integrity proof; SQL/Hudi scans over the ten physical RLI HFiles proved unsuitably coarse and require a split-aware validator. - Core Spark retry correctness is tracked independently in #19801. ### Documentation Update RFC-109 is updated in #19309 with architecture, correctness contracts, limitations, test plan, and BIGANN evidence. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
