chrevanthreddy opened a new pull request, #19291: URL: https://github.com/apache/hudi/pull/19291
## What Proposes **native approximate nearest-neighbor (ANN) vector search** for Apache Hudi, with the index stored in the Metadata Table (MDT). The base table remains the source of truth for exact vector values; the MDT holds only routing, pruning, and approximate-scoring metadata; final ranking always reads exact base-table vectors. This is a **docs-only** PR: it adds `rfc/rfc-104/rfc-104.md`, three diagrams, and the `rfc/README.md` entry. ## Core storage innovation: the posting block Instead of one MDT record per indexed vector, the index packs ~1–4K vectors into a **single MDT record** laid out column-wise (structure-of-arrays), keyed `0x10|generation|cluster|shard|blockId` so one IVF cluster forms one contiguous, prefix-scannable range. This yields: - ~1000× fewer MDT records (write/compaction/cleaner cost); - "scan a cluster" = one contiguous range read; - pay-per-promise column access (a scan pass touches only the columns it needs). §4.6 contains a small, readable end-to-end worked example (D=4, 3 vectors) showing the packed block contents and how write + query use it. ## Approach IVF routing + multibit RaBitQ (unbiased estimator with a per-vector error bound for safe pruning) + a two-pass scan (sign-plane prune → multibit refine) + RLI-arbitrated freshness + positional exact rerank + LIRE maintenance (deltas / counter-triggered compaction / local split-merge) + a generation lifecycle for zero-downtime blue/green rebuilds. Prototype measurements on a 1B-row, 128-dim table: exact-reranked recall@10 = 0.985 at nprobe=128, low single-digit second latency vs ~15s brute force. ## Scope Docs-only RFC. Tracks umbrella issue #19094; §11 maps RFC sections to the child implementation issues #19095–#19105. Resolves #19095 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
