chrevanthreddy opened a new issue, #19094: URL: https://github.com/apache/hudi/issues/19094
# RFC-104: MDT-backed vector index umbrella tracking issue ## Summary This issue tracks the upstream review and implementation breakdown for RFC-104: native vector index support in Apache Hudi. RFC-104 proposes a metadata-table-backed vector index framework for Hudi tables. The base table remains the source of truth for exact vector values. The metadata table stores vector index state used for candidate generation, query planning, and index lifecycle management. Related context: - Related: #18676 ## Scope of MVP The MVP for RFC-104 includes: - MDT-backed vector index storage - Spark-first implementation - IVF clustering and probe planning - RaBitQ approximate candidate generation - multibit RaBitQ in MVP - exact rerank from authoritative base-table vectors - create / drop / rebuild semantics - correctness and integration tests ## Non-goals for initial landing The following are not required for the first upstream landing: - additional ANN index families beyond IVF + RaBitQ - workload-specific benchmark material - deployment manifests or operational runbooks - advanced query auto-tuning - point-read / row-position exact-fetch optimizations - data-layout optimization work ## Proposed child issues - [ ] Land clean RFC for native MDT-backed vector index support - [ ] Add vector index options and Spark DDL scaffolding - [ ] Add MDT row families and payload schema for vector indexes - [ ] Implement RaBitQ encoder contract with multibit-aware scoring support - [ ] Implement Spark bootstrap for IVF clustering and MDT posting generation - [ ] Add vector index generation visibility and reader consistency semantics - [ ] Add vector query planning using centroids and cluster probing - [ ] Add MDT posting scan and approximate top-K candidate generation - [ ] Add exact rerank from authoritative base-table vectors - [ ] Add drop, rebuild, and cleanup semantics for vector indexes - [ ] Add unit, integration, and Spark SQL coverage for vector indexes ## Proposed PR order 1. RFC only 2. DDL and option scaffolding 3. MDT schema / payload model 4. RaBitQ encoder / scorer contract 5. Spark bootstrap 6. Generation visibility / consistency 7. Query planning 8. Approximate candidate generation 9. Exact rerank 10. Lifecycle and tests ## Review approach RFC-104 will be broken into small, focused PRs so the community can review: - architecture separately from code - metadata model separately from execution logic - bootstrap separately from read path - lifecycle semantics separately from scoring logic This issue acts as the umbrella tracker for that sequence. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
