This is an automated email from the ASF dual-hosted git repository.
vinothchandar pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new eccfee6dfe7d docs(rfc): RFC-109 native vector search support in Apache
Hudi (#19309)
eccfee6dfe7d is described below
commit eccfee6dfe7daf3666290e8112052f86753f979e
Author: chrevanthreddy <[email protected]>
AuthorDate: Wed Sep 9 19:18:53 2026 -0600
docs(rfc): RFC-109 native vector search support in Apache Hudi (#19309)
* docs(rfc): RFC-109 native vector search support in Apache Hudi
* docs(rfc): address vector index review feedback
* docs(rfc): keep vector consistency self-contained
* docs(rfc): define vector option defaults
* docs(rfc): define incremental vector maintenance semantics
* docs(rfc): separate vector correctness from maintenance
* docs(rfc): address vector index review feedback
* docs(rfc-109): include partition pruning in initial scope
* docs: address and clarify last round of review
* docs(rfc-109): tighten maintenance and API sections
Clarify §7 maintenance semantics in Hudi terms: table services as the
automated maintenance runtime, MDT read expectations consistent with it
being a MoR table, and vector-index correctness stated for the index
rather than as a v1 caveat.
Scope §7.3 to the built-in MDT compaction strategy for v1, with the
demand-aware vector-aware strategy called out as longer term and
dependent on query telemetry collected elsewhere. Mark §7.6 LIRE as a
future extension to revisit after v1 performance is evaluated.
Note in §8 that vector search should eventually be expressed in query
filter predicates rather than requiring a join against a TVF result.
Leaves an open VC: question in §7.2 on how to state rewrite/relocation
cost in Hudi terms.
* docs(rfc-109): postings locate rows by record key
Make the record locator concrete instead of leaving its form open. A
posting stores the record key and nothing else about where the row
lives; exact fetch resolves that key through RLI and then reads the data
file. The index stores no file group, base instant, or row position.
Merge the former S4 (row locators) into S6 (record keys), leaving five
posting-block families: S4 is now the partition dictionary needed for
partition pruning, S5 the record keys. Bootstrap still orders packing by
(cluster, fileGroup, rowPosition), but that ordering is applied at build
time rather than stored.
Record keys are stable across clustering and compaction, so a data-file
rewrite requires no vector-index update at all -- RLI alone absorbs the
relocation, as it already does for the secondary index. Drop the locator
refresh branch from the incremental write path, the locator maintenance
from MDT compaction and affinity clustering, and the large-rewrite
trigger.
Exact fetch is by key within the RLI-resolved file slice. Once RLI
populates row position, that becomes a single-page positional read;
until then the positional fast path is not available.
* Revert "docs(rfc-109): postings locate rows by record key"
This reverts commit ae4d6f5cd4393ff3816f0ec0be0f28b8cddf2570.
* docs(rfc-109): align storage model with the implementation
Correct the posting-block and key-scheme description against the vector
index implementation in #19802.
S1-S6 are sections laid end to end in one posting block record's
payload, not key-suffixed entries and not a storage-format column-family
feature. Drop the familyId suffix from the posting key, which is
0x10|generation|cluster|segment|block. Note that "family" already means
the leading key byte, so S1-S6 are not key families. The two-pass scan
therefore bounds decode work rather than bytes read, since the block is
fetched as one record.
Define the row locator where S4 is introduced: a file-group index, an
instant index, and a row position, with the indices pointing into the S5
dictionaries and the file-group dictionary carrying the partition path
used for pruning. S5 holds three dictionaries, not two.
Give the row-family table its one-byte tags, and split quantizer state
(0x02) from centroid chunks (0x03) rather than folding quantizer
metadata into the generation manifest.
Document the delta key: the posting prefix with the reserved DELTA block
id 0xFFFFFFFF followed by the record key, so one delta addresses exactly
one record and repeated writes to a record collapse on that key.
* docs(rfc-109): plainer wording for two-pass scan cost
* docs(rfc-109): rewrite in plain technical prose
The document read as a product pitch rather than an ASF design
document. Rewrite the prose without changing any design claim.
Drop the promotional framing: "one storage innovation", "the core
storage contribution of this RFC", "make results trustworthy",
"artifacted measurements", "a naive index would", and the pull-quote
callout in the abstract. State the measured recall with its caveats
rather than as a headline. Replace the named design-principle bullets
with the four constraints stated plainly.
Use Hudi's own vocabulary. A query resolves a snapshot rather than
pinning one; writers commit rather than publish; readers read rather
than consume. "RLI arbitration" becomes the RLI lookup, "the vector
hook" the vector indexer, "layer-2 skip semantics" the standard MDT
behavior for a partition a writer does not handle. Drop "contract"
where it meant behavior, and "seam" where it meant call site.
Remove restatements that the MDT commit protocol already implies,
including that generation activation is atomic and that no
intermediate step is visible.
No technical content is added or removed.
* docs(rfc-109): state relocation cost of a physical rewrite
Replace the open question in 7.2 with the answer for the storage model
as implemented. A posting stores a row locator, so clustering and
compaction both invalidate the locators of rows they move and the index
must refresh them, as the record-level index already does; the secondary
index is unaffected since it maps to record keys. Neither rewrite
requires re-routing or re-encoding.
Distinguish clustering, which is a replacecommit into new file groups,
from compaction, which rewrites a file group's base file in place at a
new instant. Rename the pure-motion optimization to relocation-only.
* docs(rfc-109): reflow section 8 preamble, fix typo
---------
Co-authored-by: Revanth Chandupatla <[email protected]>
Co-authored-by: vinoth chandar <[email protected]>
---
rfc/README.md | 6 +-
rfc/rfc-109/diagrams/01-architecture-overview.svg | 80 ++
rfc/rfc-109/diagrams/02-read-path.svg | 74 ++
rfc/rfc-109/diagrams/03-write-maintenance.svg | 74 ++
rfc/rfc-109/rfc-109.md | 1082 +++++++++++++++++++++
5 files changed, 1313 insertions(+), 3 deletions(-)
diff --git a/rfc/README.md b/rfc/README.md
index b50715599621..a92a038ba423 100644
--- a/rfc/README.md
+++ b/rfc/README.md
@@ -144,6 +144,6 @@ The list of all RFCs can be found here.
| 106 | [Record Level and Secondary Index Support for Flink
Writers](./rfc-106/rfc-106.md)
| :white_check_mark: `COMPLETED` |
| 107 | [Support data partition aware RocksDB
RecordIndexBackend](./rfc-107/rfc-107/md)
| :hammer_and_wrench: `IN PROGRESS` |
| 108 | [Multi-dataset incremental reads in Hudi
Streamer](./rfc-108/rfc-108.md)
| :eyes: `UNDER REVIEW` |
-| 109 | Hudi Native Vector Index
| :eyes:
`UNDER REVIEW` |
-| 110 | Native Full-Text Search Index
| :eyes:
`UNDER REVIEW` | | :eyes: `UNDER REVIEW` |
-| 111 | Non-blocking updates during clustering
| :eyes:
`UNDER REVIEW` |
+| 109 | [Hudi Native Vector Index](./rfc-109/rfc-109.md)
| :eyes:
`UNDER REVIEW` |
+| 110 | Native Full-Text Search Index
| :eyes:
`UNDER REVIEW` |
+| 111 | Non-blocking updates during clustering
| :eyes:
`UNDER REVIEW` |
\ No newline at end of file
diff --git a/rfc/rfc-109/diagrams/01-architecture-overview.svg
b/rfc/rfc-109/diagrams/01-architecture-overview.svg
new file mode 100644
index 000000000000..98c09cfcdc55
--- /dev/null
+++ b/rfc/rfc-109/diagrams/01-architecture-overview.svg
@@ -0,0 +1,80 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0.
+-->
+<svg xmlns="http://www.w3.org/2000/svg"; width="1160" height="650" viewBox="0 0
1160 650" role="img" aria-labelledby="title desc">
+ <title id="title">RFC-109 MDT-native vector index architecture</title>
+ <desc id="desc">Base table stores exact vectors. Metadata table stores
centroids, quantizer metadata, cluster manifests, and packed posting blocks.
Queries use MDT first and exact re-rank from base table vectors.</desc>
+ <defs>
+ <filter id="shadow" x="-10%" y="-10%" width="120%" height="120%">
+ <feDropShadow dx="0" dy="3" stdDeviation="3" flood-opacity="0.18"/>
+ </filter>
+ <marker id="arrow" markerWidth="10" markerHeight="10" refX="8" refY="3"
orient="auto" markerUnits="strokeWidth">
+ <path d="M0,0 L0,6 L9,3 z" fill="#334155"/>
+ </marker>
+ </defs>
+
+ <rect width="1160" height="650" fill="#f8fafc"/>
+ <text x="580" y="38" text-anchor="middle" font-family="Arial, sans-serif"
font-size="28" font-weight="700" fill="#0f172a">RFC-109: MDT-native Vector
Search</text>
+ <text x="580" y="66" text-anchor="middle" font-family="Arial, sans-serif"
font-size="15" fill="#475569">Base table keeps exact VECTOR(D); MDT stores IVF
+ RaBitQ routing and posting metadata</text>
+
+ <g transform="translate(55,105)" filter="url(#shadow)">
+ <rect width="430" height="300" rx="14" fill="#dbeafe" stroke="#2563eb"
stroke-width="2"/>
+ <rect width="430" height="48" rx="14" fill="#2563eb"/>
+ <text x="215" y="31" text-anchor="middle" font-family="Arial, sans-serif"
font-size="18" font-weight="700" fill="white">Main Hudi Table</text>
+ <g font-family="Arial, sans-serif" font-size="15" fill="#1e3a8a">
+ <text x="30" y="88" font-weight="700">Parquet / log file groups</text>
+ <text x="50" y="122">_hoodie_record_key</text>
+ <text x="50" y="152">_hoodie_partition_path</text>
+ <text x="50" y="182">user columns</text>
+ <rect x="45" y="210" width="340" height="54" rx="8" fill="#eff6ff"
stroke="#60a5fa"/>
+ <text x="215" y="233" text-anchor="middle" font-weight="700">embedding
VECTOR(D)</text>
+ <text x="215" y="254" text-anchor="middle" font-size="13">authoritative
exact vector values</text>
+ </g>
+ </g>
+
+ <g transform="translate(675,105)" filter="url(#shadow)">
+ <rect width="430" height="300" rx="14" fill="#dcfce7" stroke="#16a34a"
stroke-width="2"/>
+ <rect width="430" height="48" rx="14" fill="#16a34a"/>
+ <text x="215" y="31" text-anchor="middle" font-family="Arial, sans-serif"
font-size="18" font-weight="700" fill="white">MDT Partition:
vector_index_<name></text>
+ <g font-family="Arial, sans-serif" font-size="14" fill="#14532d">
+ <rect x="24" y="72" width="178" height="72" rx="8" fill="white"
stroke="#86efac"/>
+ <text x="36" y="96" font-weight="700">Index metadata</text>
+ <text x="36" y="119">__manifest__</text>
+ <text x="36" y="137">persisted-format version</text>
+
+ <rect x="228" y="72" width="178" height="72" rx="8" fill="white"
stroke="#86efac"/>
+ <text x="240" y="96" font-weight="700">Generation rows</text>
+ <text x="240" y="119">M|generation</text>
+ <text x="240" y="137">T|centroids, C|clusters, F|markers</text>
+
+ <rect x="24" y="170" width="382" height="88" rx="8" fill="#fef3c7"
stroke="#f59e0b" stroke-width="2"/>
+ <text x="36" y="196" font-weight="700" fill="#92400e">Posting
blocks</text>
+ <text x="36" y="221"
fill="#92400e">P|generation|cluster|shard|blockId</text>
+ <text x="36" y="244" fill="#92400e">packed codes, factors, locators,
record keys</text>
+ </g>
+ </g>
+
+ <path d="M488 248 C555 225, 606 225, 671 248" fill="none" stroke="#334155"
stroke-width="3" marker-end="url(#arrow)"/>
+ <text x="580" y="215" text-anchor="middle" font-family="Arial, sans-serif"
font-size="14" fill="#334155">bootstrap / writes create index metadata</text>
+
+ <path d="M671 309 C598 365, 556 365, 488 309" fill="none" stroke="#334155"
stroke-width="3" marker-end="url(#arrow)"/>
+ <text x="580" y="386" text-anchor="middle" font-family="Arial, sans-serif"
font-size="14" fill="#334155">queries use MDT candidates, then read exact
vectors</text>
+
+ <g transform="translate(80,470)" font-family="Arial, sans-serif"
filter="url(#shadow)">
+ <rect width="1000" height="118" rx="14" fill="#fff7ed" stroke="#f97316"
stroke-width="2"/>
+ <text x="500" y="32" text-anchor="middle" font-size="17" font-weight="700"
fill="#9a3412">Query flow</text>
+ <g font-size="13" fill="#334155">
+ <circle cx="80" cy="76" r="22" fill="#2563eb"/><text x="80" y="81"
text-anchor="middle" fill="white" font-weight="700">1</text>
+ <text x="120" y="80">load manifest + centroids</text>
+ <circle cx="320" cy="76" r="22" fill="#2563eb"/><text x="320" y="81"
text-anchor="middle" fill="white" font-weight="700">2</text>
+ <text x="360" y="80">probe IVF clusters</text>
+ <circle cx="540" cy="76" r="22" fill="#2563eb"/><text x="540" y="81"
text-anchor="middle" fill="white" font-weight="700">3</text>
+ <text x="580" y="80">scan postings + RaBitQ score</text>
+ <circle cx="800" cy="76" r="22" fill="#2563eb"/><text x="800" y="81"
text-anchor="middle" fill="white" font-weight="700">4</text>
+ <text x="840" y="80">exact re-rank from base table</text>
+ </g>
+ </g>
+</svg>
diff --git a/rfc/rfc-109/diagrams/02-read-path.svg
b/rfc/rfc-109/diagrams/02-read-path.svg
new file mode 100644
index 000000000000..a28d8d8f565f
--- /dev/null
+++ b/rfc/rfc-109/diagrams/02-read-path.svg
@@ -0,0 +1,74 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0.
+-->
+<svg xmlns="http://www.w3.org/2000/svg"; width="1160" height="560" viewBox="0 0
1160 560" role="img" aria-labelledby="title desc">
+ <title id="title">RFC-109 vector search read path</title>
+ <desc id="desc">Read path loads vector metadata, probes centroids, scans
postings, scores RaBitQ codes, reads base vectors, and exact reranks
candidates.</desc>
+ <defs>
+ <filter id="shadow" x="-10%" y="-10%" width="120%"
height="120%"><feDropShadow dx="0" dy="3" stdDeviation="3"
flood-opacity="0.18"/></filter>
+ <marker id="arrow" markerWidth="10" markerHeight="10" refX="8" refY="3"
orient="auto" markerUnits="strokeWidth"><path d="M0,0 L0,6 L9,3 z"
fill="#334155"/></marker>
+ </defs>
+ <rect width="1160" height="560" fill="#f8fafc"/>
+ <text x="580" y="38" text-anchor="middle" font-family="Arial, sans-serif"
font-size="28" font-weight="700" fill="#0f172a">Vector Query Read Path</text>
+ <text x="580" y="66" text-anchor="middle" font-family="Arial, sans-serif"
font-size="15" fill="#475569">MDT narrows candidates; exact ranking uses
base-table VECTOR(D)</text>
+
+ <g font-family="Arial, sans-serif" filter="url(#shadow)">
+ <rect x="40" y="125" width="140" height="82" rx="12" fill="#dbeafe"
stroke="#2563eb" stroke-width="2"/>
+ <text x="110" y="154" text-anchor="middle" font-size="15"
font-weight="700" fill="#1e40af">Query vector</text>
+ <text x="110" y="181" text-anchor="middle" font-size="13" fill="#1e40af">q
VECTOR(D)</text>
+
+ <rect x="225" y="112" width="160" height="108" rx="12" fill="#dcfce7"
stroke="#16a34a" stroke-width="2"/>
+ <text x="305" y="141" text-anchor="middle" font-size="15"
font-weight="700" fill="#14532d">Metadata cache</text>
+ <text x="305" y="166" text-anchor="middle" font-size="12"
fill="#14532d">manifest</text>
+ <text x="305" y="184" text-anchor="middle" font-size="12"
fill="#14532d">centroids</text>
+ <text x="305" y="202" text-anchor="middle" font-size="12"
fill="#14532d">quantizer + clusters</text>
+
+ <rect x="430" y="112" width="155" height="108" rx="12" fill="#eef2ff"
stroke="#4f46e5" stroke-width="2"/>
+ <text x="508" y="141" text-anchor="middle" font-size="15"
font-weight="700" fill="#3730a3">Probe IVF</text>
+ <text x="508" y="169" text-anchor="middle" font-size="12"
fill="#3730a3">query vs K centroids</text>
+ <text x="508" y="191" text-anchor="middle" font-size="12"
fill="#3730a3">select top-P clusters</text>
+
+ <rect x="630" y="112" width="180" height="108" rx="12" fill="#fef3c7"
stroke="#f59e0b" stroke-width="2"/>
+ <text x="720" y="141" text-anchor="middle" font-size="15"
font-weight="700" fill="#92400e">Scan postings</text>
+ <text x="720" y="169" text-anchor="middle" font-size="12"
fill="#92400e">P|G|cluster|shard|*</text>
+ <text x="720" y="191" text-anchor="middle" font-size="12"
fill="#92400e">selected cluster shards</text>
+
+ <rect x="855" y="112" width="150" height="108" rx="12" fill="#fae8ff"
stroke="#a855f7" stroke-width="2"/>
+ <text x="930" y="141" text-anchor="middle" font-size="15"
font-weight="700" fill="#6b21a8">RaBitQ score</text>
+ <text x="930" y="169" text-anchor="middle" font-size="12"
fill="#6b21a8">popcount XOR</text>
+ <text x="930" y="191" text-anchor="middle" font-size="12"
fill="#6b21a8">top-R shortlist</text>
+
+ <path d="M180 166 H222" stroke="#334155" stroke-width="3"
marker-end="url(#arrow)"/>
+ <path d="M385 166 H427" stroke="#334155" stroke-width="3"
marker-end="url(#arrow)"/>
+ <path d="M585 166 H627" stroke="#334155" stroke-width="3"
marker-end="url(#arrow)"/>
+ <path d="M810 166 H852" stroke="#334155" stroke-width="3"
marker-end="url(#arrow)"/>
+ </g>
+
+ <g font-family="Arial, sans-serif" filter="url(#shadow)">
+ <rect x="180" y="315" width="270" height="100" rx="12" fill="#dbeafe"
stroke="#2563eb" stroke-width="2"/>
+ <text x="315" y="345" text-anchor="middle" font-size="16"
font-weight="700" fill="#1e40af">Read candidate rows</text>
+ <text x="315" y="373" text-anchor="middle" font-size="13"
fill="#1e40af">fileGroupId + partitionPath from postings</text>
+ <text x="315" y="395" text-anchor="middle" font-size="13"
fill="#1e40af">fetch original VECTOR(D)</text>
+
+ <rect x="520" y="315" width="220" height="100" rx="12" fill="#dcfce7"
stroke="#16a34a" stroke-width="2"/>
+ <text x="630" y="345" text-anchor="middle" font-size="16"
font-weight="700" fill="#14532d">Exact re-rank</text>
+ <text x="630" y="373" text-anchor="middle" font-size="13"
fill="#14532d">compute exact distance</text>
+ <text x="630" y="395" text-anchor="middle" font-size="13"
fill="#14532d">sort candidates</text>
+
+ <rect x="810" y="315" width="150" height="100" rx="12" fill="#fee2e2"
stroke="#ef4444" stroke-width="2"/>
+ <text x="885" y="354" text-anchor="middle" font-size="16"
font-weight="700" fill="#991b1b">Top-K</text>
+ <text x="885" y="382" text-anchor="middle" font-size="13"
fill="#991b1b">final results</text>
+
+ <path d="M930 220 C930 270, 315 270, 315 312" fill="none" stroke="#334155"
stroke-width="3" marker-end="url(#arrow)"/>
+ <path d="M450 365 H517" stroke="#334155" stroke-width="3"
marker-end="url(#arrow)"/>
+ <path d="M740 365 H807" stroke="#334155" stroke-width="3"
marker-end="url(#arrow)"/>
+ </g>
+
+ <g transform="translate(90,470)" font-family="Arial, sans-serif">
+ <rect width="980" height="48" rx="10" fill="#e2e8f0"/>
+ <text x="490" y="30" text-anchor="middle" font-size="14"
fill="#334155">Batched queries group by selected cluster/shard so each posting
range is scanned once and scored for multiple query vectors.</text>
+ </g>
+</svg>
diff --git a/rfc/rfc-109/diagrams/03-write-maintenance.svg
b/rfc/rfc-109/diagrams/03-write-maintenance.svg
new file mode 100644
index 000000000000..1e1cd398c47d
--- /dev/null
+++ b/rfc/rfc-109/diagrams/03-write-maintenance.svg
@@ -0,0 +1,74 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0.
+-->
+<svg xmlns="http://www.w3.org/2000/svg"; width="1160" height="620" viewBox="0 0
1160 620" role="img" aria-labelledby="title desc">
+ <title id="title">RFC-109 vector index bootstrap write and maintenance
path</title>
+ <desc id="desc">Spark bootstrap trains centroids and writes a complete
generation. Each data-write instant writes a freshness marker and conditionally
writes posting deltas. LIRE maintenance re-encodes affected clusters and
atomically publishes a complete generation.</desc>
+ <defs>
+ <filter id="shadow" x="-10%" y="-10%" width="120%"
height="120%"><feDropShadow dx="0" dy="3" stdDeviation="3"
flood-opacity="0.18"/></filter>
+ <marker id="arrow" markerWidth="10" markerHeight="10" refX="8" refY="3"
orient="auto" markerUnits="strokeWidth"><path d="M0,0 L0,6 L9,3 z"
fill="#334155"/></marker>
+ </defs>
+ <rect width="1160" height="620" fill="#f8fafc"/>
+ <text x="580" y="38" text-anchor="middle" font-family="Arial, sans-serif"
font-size="28" font-weight="700" fill="#0f172a">Bootstrap, Writes, and LIRE
Maintenance</text>
+ <text x="580" y="66" text-anchor="middle" font-family="Arial, sans-serif"
font-size="15" fill="#475569">Spark builds and refreshes MDT vector generations
without storing index payloads in base files</text>
+
+ <g font-family="Arial, sans-serif" filter="url(#shadow)">
+ <text x="70" y="118" font-size="18" font-weight="700"
fill="#1e40af">Bootstrap / rebuild</text>
+ <rect x="70" y="140" width="160" height="78" rx="12" fill="#dbeafe"
stroke="#2563eb" stroke-width="2"/>
+ <text x="150" y="171" text-anchor="middle" font-size="14"
font-weight="700" fill="#1e40af">Read snapshot</text>
+ <text x="150" y="194" text-anchor="middle" font-size="12"
fill="#1e40af">record + VECTOR(D)</text>
+ <rect x="290" y="140" width="160" height="78" rx="12" fill="#eef2ff"
stroke="#4f46e5" stroke-width="2"/>
+ <text x="370" y="171" text-anchor="middle" font-size="14"
font-weight="700" fill="#3730a3">KMeans</text>
+ <text x="370" y="194" text-anchor="middle" font-size="12"
fill="#3730a3">train IVF centroids</text>
+ <rect x="510" y="140" width="180" height="78" rx="12" fill="#fef3c7"
stroke="#f59e0b" stroke-width="2"/>
+ <text x="600" y="171" text-anchor="middle" font-size="14"
font-weight="700" fill="#92400e">Assign + RaBitQ</text>
+ <text x="600" y="194" text-anchor="middle" font-size="12"
fill="#92400e">cluster, shard, code</text>
+ <rect x="750" y="140" width="230" height="78" rx="12" fill="#dcfce7"
stroke="#16a34a" stroke-width="2"/>
+ <text x="865" y="171" text-anchor="middle" font-size="14"
font-weight="700" fill="#14532d">Write MDT generation</text>
+ <text x="865" y="194" text-anchor="middle" font-size="12"
fill="#14532d">M|, T| chunks, C|, P|, F|</text>
+ <path d="M230 179 H287" stroke="#334155" stroke-width="3"
marker-end="url(#arrow)"/>
+ <path d="M450 179 H507" stroke="#334155" stroke-width="3"
marker-end="url(#arrow)"/>
+ <path d="M690 179 H747" stroke="#334155" stroke-width="3"
marker-end="url(#arrow)"/>
+ </g>
+
+ <g font-family="Arial, sans-serif" filter="url(#shadow)">
+ <text x="70" y="288" font-size="18" font-weight="700"
fill="#9a3412">Incremental data-write maintenance</text>
+ <rect x="70" y="310" width="185" height="78" rx="12" fill="#fff7ed"
stroke="#f97316" stroke-width="2"/>
+ <text x="162" y="341" text-anchor="middle" font-size="14"
font-weight="700" fill="#9a3412">Data-write instant</text>
+ <text x="162" y="364" text-anchor="middle" font-size="12"
fill="#9a3412">rows, delete, or no-op</text>
+ <rect x="315" y="310" width="185" height="78" rx="12" fill="#fef3c7"
stroke="#f59e0b" stroke-width="2"/>
+ <text x="407" y="341" text-anchor="middle" font-size="14"
font-weight="700" fill="#92400e">Encode if needed</text>
+ <text x="407" y="364" text-anchor="middle" font-size="12"
fill="#92400e">P| delta is conditional</text>
+ <rect x="560" y="310" width="220" height="78" rx="12" fill="#dcfce7"
stroke="#16a34a" stroke-width="2"/>
+ <text x="670" y="341" text-anchor="middle" font-size="14"
font-weight="700" fill="#14532d">Write F| marker</text>
+ <text x="670" y="364" text-anchor="middle" font-size="12" fill="#14532d">+
optional P| in same MDT commit</text>
+ <rect x="840" y="310" width="190" height="78" rx="12" fill="#fee2e2"
stroke="#ef4444" stroke-width="2"/>
+ <text x="935" y="341" text-anchor="middle" font-size="14"
font-weight="700" fill="#991b1b">Reader invariant</text>
+ <text x="935" y="364" text-anchor="middle" font-size="12"
fill="#991b1b">contiguous marker frontier</text>
+ <path d="M255 349 H312" stroke="#334155" stroke-width="3"
marker-end="url(#arrow)"/>
+ <path d="M500 349 H557" stroke="#334155" stroke-width="3"
marker-end="url(#arrow)"/>
+ <path d="M780 349 H837" stroke="#334155" stroke-width="3"
marker-end="url(#arrow)"/>
+ </g>
+
+ <g font-family="Arial, sans-serif" filter="url(#shadow)">
+ <text x="70" y="458" font-size="18" font-weight="700"
fill="#14532d">LIRE-style maintenance</text>
+ <rect x="70" y="480" width="190" height="78" rx="12" fill="#dcfce7"
stroke="#16a34a" stroke-width="2"/>
+ <text x="165" y="511" text-anchor="middle" font-size="14"
font-weight="700" fill="#14532d">Detect drift</text>
+ <text x="165" y="534" text-anchor="middle" font-size="12"
fill="#14532d">imbalance, NPA, clustering</text>
+ <rect x="320" y="480" width="210" height="78" rx="12" fill="#f0fdf4"
stroke="#22c55e" stroke-width="2"/>
+ <text x="425" y="511" text-anchor="middle" font-size="14"
font-weight="700" fill="#14532d">Scan affected ranges</text>
+ <text x="425" y="534" text-anchor="middle" font-size="12"
fill="#14532d">C|cluster and P|cluster|*</text>
+ <rect x="590" y="480" width="210" height="78" rx="12" fill="#fef3c7"
stroke="#f59e0b" stroke-width="2"/>
+ <text x="695" y="511" text-anchor="middle" font-size="14"
font-weight="700" fill="#92400e">Refresh metadata</text>
+ <text x="695" y="534" text-anchor="middle" font-size="12"
fill="#92400e">T| centroids, C|, P| blocks</text>
+ <rect x="860" y="480" width="210" height="78" rx="12" fill="#dcfce7"
stroke="#16a34a" stroke-width="2"/>
+ <text x="965" y="511" text-anchor="middle" font-size="14"
font-weight="700" fill="#14532d">Publish visibility</text>
+ <text x="965" y="534" text-anchor="middle" font-size="12"
fill="#14532d">atomic generation publish</text>
+ <path d="M260 519 H317" stroke="#334155" stroke-width="3"
marker-end="url(#arrow)"/>
+ <path d="M530 519 H587" stroke="#334155" stroke-width="3"
marker-end="url(#arrow)"/>
+ <path d="M800 519 H857" stroke="#334155" stroke-width="3"
marker-end="url(#arrow)"/>
+ </g>
+</svg>
diff --git a/rfc/rfc-109/rfc-109.md b/rfc/rfc-109/rfc-109.md
new file mode 100644
index 000000000000..6646449f043f
--- /dev/null
+++ b/rfc/rfc-109/rfc-109.md
@@ -0,0 +1,1082 @@
+```html
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+```
+
+# RFC-109: Native Vector Search Support in Apache Hudi
+
+## Proposers
+
+@chrevanthreddy
+
+## Approvers
+
+@vinothchandar
+
+## Status
+
+Umbrella issue:
[apache/hudi#19094](https://github.com/apache/hudi/issues/19094)
+
+Related: [apache/hudi#18676](https://github.com/apache/hudi/issues/18676)
+
+State: UNDER REVIEW
+
+---
+
+## Table of Contents
+
+- [Abstract](#abstract)
+- [1. Goals and Non-Goals](#1-goals-and-non-goals)
+- [Limitations](#limitations)
+- [2. Architecture](#2-architecture)
+- [3. IVF + RaBitQ Index Algorithm](#3-ivf--rabitq-index-algorithm)
+- [4. Metadata Table Storage Model: the Posting
Block](#4-metadata-table-storage-model-the-posting-block)
+- [5. Bootstrap and Write Path](#5-bootstrap-and-write-path)
+- [6. Read Path](#6-read-path)
+- [7. Maintenance, Rebalancing, and
Cleaner](#7-maintenance-rebalancing-and-cleaner)
+- [8. Spark API Surface](#8-spark-api-surface)
+- [9. Correctness, Compatibility, and
Freshness](#9-correctness-compatibility-and-freshness)
+- [10. Test Plan](#10-test-plan)
+- [11. Rollout and MVP Scope](#11-rollout-and-mvp-scope)
+- [12. References](#12-references)
+- [Appendix A. Controlled BIGANN Research
Evidence](#appendix-a-controlled-bigann-research-evidence)
+
+---
+
+## Abstract
+
+This RFC proposes approximate nearest-neighbor (ANN) vector search over Hudi's
fixed-dimension
+`VECTOR(D[, elementType])` logical type, so that a query for the K rows
nearest a given vector
+can be answered without scanning the table or copying data into a separate
system.
+
+The only option on a Hudi table today is a brute-force scan: read every vector
and compute every
+distance. That is exact, but linear in table size — tens of seconds at a
billion rows. This RFC
+adds an index that reads a bounded fraction of the index and the table, and
stays consistent with
+the table under upserts and deletes. The index is stored in the Hudi Metadata
Table (MDT)
+alongside the record-level and secondary indexes, and is maintained by the
same table services.
+
+The design uses IVF clustering to decide which vectors to examine, RaBitQ
quantization to compare
+them cheaply, and exact re-ranking against base-table vectors to order the
result. Two terms
+recur. A **posting** is the index entry for one record: its cluster
assignment, compressed scoring
+material, record key, and row locator. A **generation** is one complete index
state — vector
+schema, centroids, quantizer, routing, postings, and freshness frontier —
activated by a single
+MDT commit. Centroids, quantizer, and routing are fixed once a generation is
active; postings and
+the frontier advance through further MDT commits within it.
+
+Rather than one MDT record per posting, the index packs \~1–4K vectors into a
**posting block**
+laid out as parallel arrays, keyed so that one IVF cluster occupies a
contiguous, prefix-scannable
+key range (§4). The base table remains authoritative for vector values; the
MDT holds only
+routing, pruning, and approximate-scoring metadata.
+
+Measured on a 1-billion-row, 128-dimensional table, approximate-only
recall\@10 is 0.822, 0.857,
+and 0.871 at `nprobe` 16, 32, and 64; exact reranking reaches 0.960 at
`nprobe=32` with
+`refineFactor=50` (Appendix A). Approximate-only is the lower-latency mode;
exact reranking is the
+higher-quality mode. Measurement caveats are recorded with the results.
+
+---
+
+## 1. Goals and Non-Goals
+
+### 1.1 Goals
+
+1. Keep authoritative vector values in a top-level base-table `VECTOR(D[,
elementType])` column.
+2. Store the vector index in the MDT, maintained by Hudi metadata-table
commits, compaction,
+ and cleaning — no hidden or generated columns in base-table files.
+3. Bound candidate discovery: prune to the requested table partitions, probe a
limited number of
+ clusters, scan contiguous key ranges, and score compressed codes under a
pruning bound.
+4. Select candidates approximately, but order them by exact distance on
base-table vectors.
+5. Stay transactionally consistent: all reads for a query resolved against one
table snapshot,
+ with defined behavior under inserts, updates, deletes, and clustering.
+6. Be engine-neutral in design; Spark is the first implementation.
+7. Maintain the index incrementally (no global rebuild for normal churn) and
support
+ versioned, zero-downtime rebuilds.
+
+### 1.2 Non-Goals (initial landing)
+
+- ANN families beyond IVF + RaBitQ (e.g. HNSW, DiskANN).
+- Filtered search (arbitrary predicate + kNN) as a first-class planned
operation.
+- Time-travel-consistent index reads for historical snapshots.
+- Native non-Spark generation construction, GPU encoding, and
workload-specific auto-tuning.
+
+### 1.3 Alternatives considered
+
+- **One index record per vector** is simple but creates billions of MDT
records and excessive
+ write, compaction, and scan amplification; posting blocks preserve MDT
ownership while
+ amortizing that overhead.
+- **Dedicated index files in the table** permit specialized layouts but
introduce a second
+ commit, cleaning, and snapshot protocol. MDT records reuse Hudi's existing
transaction and
+ table-service machinery.
+- **External sidecar indexes or vector databases** may offer richer serving
features but lose
+ atomic Hudi snapshot semantics and require another storage system. They
remain valid when
+ independent serving infrastructure is desired.
+- **Native ANN libraries** improve local kernels but do not define durable
object-store layout,
+ multi-writer maintenance, or snapshot visibility. They can be used behind
the interfaces in
+ this RFC without changing what is written to storage.
+
+---
+
+## Limitations
+
+The first implementation fails fast, or falls back to an exact scan, in the
following cases:
+
+- **Vector schema evolution.** Changing the indexed vector's dimension or
element type makes
+ the active generation incompatible. Existing-index maintenance and indexed
queries fail
+ closed after such a change; rebuilding or recreating the index establishes
and atomically
+ activates a compatible generation. This RFC does not claim transparent
online migration
+ across incompatible vector schemas.
+- **Filtered ANN.** The initial implementation is intended to support
partition pruning: posting
+ dictionaries preserve partition paths so the query path can exclude postings
outside the
+ requested table partitions before candidate-heap selection and exact
reranking. This produces
+ partition-local top-K rather than filtering a global top-K after search.
Arbitrary
+ predicate-plus-kNN planning over non-partition columns is not part of the
initial implementation.
+- **Index quality.** IVF quality depends on the training sample and corpus
geometry. A fixed
+ seed makes fitting repeatable for fixed inputs and partitioning, but does
not make centroids
+ invariant to changed file/RDD partition layouts. Recall must therefore be
measured for each
+ built generation.
+- **Local incremental rebalancing.** LIRE split/merge is design-specified but
not part of the
+ first implementation. Until it lands, sustained skew or centroid drift is
corrected by a
+ full generation rebuild.
+- **Maintenance scale evidence.** Bootstrap and query paths are measured at
one billion rows.
+ Incremental COW/MoR mutation, compaction/clustering relocation, and
multi-writer races require
+ the test evidence in §10 before their implementations are declared complete.
+- **MoR exact-fetch cost.** Log-resident finalists are resolved by key through
the merged file
+ slice at the query's snapshot. This is correct but can cost more than a
base-file
+ positional read, and has not been measured at scale.
+
+---
+
+## 2. Architecture
+
+
+
+The design splits responsibilities the way Hudi already does between the data
table and the
+metadata table:
+
+```text
+DATA TABLE (parquet/orc) METADATA TABLE (vector_index partition)
+ authoritative vectors + payload ←── the index: centroids, quantizer,
posting blocks,
+ read only for final re-ranking cluster manifests, generation
manifest
+ read for candidate generation
+```
+
+### 2.1 Design constraints
+
+Four constraints shape the rest of this document.
+
+Correctness does not depend on table services having run. Delta postings and
the RLI lookup make
+a completed update discoverable, and prevent an older version of a record from
being returned,
+before compaction or clustering runs. Exact reranking reads MoR log-resident
finalists from the
+merged file slice; compaction reduces that read cost but is not a prerequisite
for a record to be
+found, materialized, or superseded.
+
+Queries do not write. The query path writes neither the MDT nor the data
table. Workload signals
+such as log-resident finalist count, stale-candidate crowding, and fetch-key
mismatch leave only
+as metrics. Maintenance may read an externally aggregated, decayed form of
those signals, and
+must behave correctly when they are absent.
+
+A query resolves the table snapshot and the ACTIVE generation once, at
planning. Every subsequent
+read it makes — MDT, RLI, file slice, base table, routing, cached centroids —
uses that pair.
+
+The index is derived state. The data table remains authoritative, so index
files can be compacted,
+rebalanced, rebuilt, or dropped without changing table data.
+
+Each vector index is one MDT partition. RFC-109 adds no table properties,
common writer-
+dispatch changes, timeline-semantics changes, or behavior visible to
non-vector readers or
+writers. Every consistency mechanism is either a record in the vector-index
MDT partition or
+logic in the RFC-owned vector indexer and query planner. Creating an index:
+
+```sql
+CREATE INDEX embedding_idx
+ON products
+USING VECTOR (embedding)
+OPTIONS (
+ 'vector.metric' = 'cosine',
+ 'vector.quantizer' = 'IVF_RABITQ',
+ 'vector.num_clusters'= '4096'
+);
+```
+
+creates:
+
+```text
+.hoodie/metadata/vector_index_embedding_idx/
+```
+
+and does not add generated columns to the base-table schema. RFC-109 uses the
RFC-99 vector type
+as defined: the source column must be a top-level `VECTOR(D[, elementType])`.
The table schema is
+authoritative for `D` and element type; index definitions and generation
manifests repeat them
+only so a mismatch can be detected.
+
+The current RFC-99 storage backing is fixed-width bytes: Avro `FIXED` and
Parquet
+`FIXED_LEN_BYTE_ARRAY(D × elementWidth)`. Engine adapters expose idiomatic
values—Spark uses
+an annotated `ArrayType(FloatType|DoubleType|ByteType)`—and convert at the
storage boundary.
+This fixed width also avoids Parquet LIST repetition-level traversal during
positional exact
+fetch. A plain `ARRAY<FLOAT>` is not implicitly indexable and requires
explicit migration or
+backfill to `VECTOR(D)`; index creation neither reinterprets nor rewrites it.
+
+The query path uses the MDT first to discover candidates, then reads
base-table vectors for
+exact re-ranking:
+
+```text
+query vector
+ → compare to centroids, pick nprobe clusters (in-memory, ms)
+ → MDT prefix-scan those clusters' posting blocks (targeted range reads)
+ → two-pass RaBitQ scoring, keep refineFactor·K best (bit math + error
bounds)
+ → validate candidate freshness via Record Level Index (batched point
lookups)
+ → fetch ONLY those rows from the base table by position (page-level reads)
+ → exact distance on real vectors → final top-K
+```
+
+---
+
+## 3. IVF + RaBitQ Index Algorithm
+
+An ANN index has to decide which vectors to examine, and how to compare them
without
+full-precision arithmetic on everything examined. IVF answers the first,
RaBitQ the second.
+
+### 3.1 Candidate selection: IVF routing
+
+Inverted File (IVF) indexing clusters vectors with KMeans into `numClusters`
groups (e.g.
+\~4K–64K). Each vector belongs to its nearest centroid. A query compares
against the centroids
+only (thousands, not billions), selects the `nprobe` nearest clusters, and
scans only those
+clusters' entries. `nprobe` is the recall dial.
+
+IVF suits a lakehouse-resident index because a cluster's entries can be stored
contiguously,
+which maps onto sorted key ranges in the MDT (§4). Graph indexes such as HNSW
reach higher recall
+in memory but require random traversal of the whole graph, which does not fit
columnar, immutable,
+object-store files.
+
+### 3.2 Comparison: RaBitQ quantization
+
+A probed cluster still holds thousands of full vectors. Quantization stores a
small *code* per
+vector plus a few correction scalars, so most comparisons run on the codes and
only the best few
+hundred candidates are re-checked against the actual vectors.
+
+RaBitQ is used instead of scalar (SQ), product (PQ), or plain binary
quantization for four
+reasons:
+
+1. Each code carries scalars that turn a bit-level dot product into an
unbiased estimate of the
+ true distance, together with a bound on the estimate's error. The bound
permits pruning
+ without loss: a vector is skipped only when even its best possible distance
cannot reach
+ top-K.
+2. No codebooks. RaBitQ requires only a random rotation, given by a seed, and
the centroids. Both
+ are small and both are versioned in the index metadata, so nothing has to
be retrained as data
+ drifts.
+3. Precision is a parameter. B = 1 bit/dim is a coarse filter; B = 4 bits/dim
approaches SQ
+ quality at roughly 8× less space. The two-pass scan (§6.2) uses both.
+4. One stored code serves L2, cosine, and dot-product; the metric is applied
at query time.
+
+The encoding has four parts:
+
+- **Residual.** Each vector is stored as its difference from its centroid, `r
= v − c`. Residuals
+ are smaller and centered, so fewer bits suffice.
+- **Rotation.** One fixed random orthonormal rotation `R`, derived from a
per-generation seed, is
+ applied first: `x = R·r`. This distributes information evenly across
dimensions, which is the
+ condition under which the error bound holds for arbitrary data. Only the
seed is stored.
+- **Code.** `x` is quantized to B bits per dimension and stored as bit planes:
plane 0 holds bit 0
+ of every dimension, plane 1 holds bit 1, and so on. The top plane alone is a
1-bit sign sketch.
+ Scoring one plane against the transformed query is `AND`/`XOR` plus
`popcount`, a few
+ instructions per 64 dimensions.
+- **Factors.** A few scalars per vector — residual norm, two rescale factors,
an error term, a
+ centroid correction — that convert the plane arithmetic into the distance
estimate and its
+ interval.
+
+At query time the query vector is transformed the same way (`R·(q − c)` per
probed cluster),
+and the per-plane popcounts combine with the stored factors into the estimate
and its bound.
+The estimate builds the shortlist; exact base-table distances produce the
final ranking (§6).
+
+### 3.3 Why this fits Hudi
+
+- Centroids are small enough to load at planning time: `K × D` floats (\~12 MB
for K=4096,
+ D=768).
+- Codes are compact: a 1B × 128-dim float table's raw vectors are \~512 GB;
the RaBitQ index
+ including keys and locators is \~136 GB, and the scanned portion per query
is tens of MB.
+- Posting keys are prefix-scannable by generation, cluster, and segment (§4).
+- Quantizer state is stable: a seed and centroids, no per-generation learned
codebook.
+- Exact re-ranking preserves correctness for returned candidates.
+
+---
+
+## 4. Metadata Table Storage Model: the Posting Block
+
+### 4.1 The posting block
+
+Writing one MDT record per indexed vector would mean a billion MDT records per
generation on a
+billion-row table, which is impractical to write, compact, scan, and clean.
Instead, the index
+sorts entries by `(cluster, fileGroup, rowPosition)` and packs
+\~1–4K vectors' worth of codes and metadata into a single **posting block**,
laid out
+column-wise (structure-of-arrays) so each scan pass touches only the columns
it needs:
+
+```text
+POSTING BLOCK (~512 KB target)
+┌───────────────────────────────────────────────┐
+│ S1 sign planes ← pass 1 touches this │
+│ S2 extra bit planes ← pass 2, survivors │
+│ S3 factor arrays ← both passes │
+│ S4 row locators ← finalists only │
+│ S5 dicts (file groups, instants, partitions) │
+│ S6 record keys ← finalists only │
+└───────────────────────────────────────────────┘
+key: 0x10 | generation | clusterId | segmentId | blockId
+```
+
+`S1`–`S6` label the six arrays laid end to end in one posting block's payload.
They are not
+separate MDT records and they do not use any storage-format column-family
feature: a block is a
+single record under a single key, and a scan pass seeks to the byte range of
the arrays it needs
+and leaves the rest undecoded. Because the layout is expressed as offsets
within the payload
+rather than as a format capability, it is portable to any base file format the
MDT can use.
+
+Note the word *family* is already taken in the key scheme: there it means the
leading byte that
+separates manifests, quantizer, centroids, cluster stats, markers, and
postings from each other
+(§4.2). `S1`–`S6` are sections inside one posting record, not key families.
+
+**Row locator.** `S4` stores one fixed-width row locator per vector: a
file-group index, an
+instant index, and a row position, where the two indices point into the `S5`
dictionaries. The
+file-group dictionary carries each file group's partition path alongside it,
which is what lets
+the query path prune by partition before scoring (see Limitations). Locators
are hints — §6.3
+validates the finalist's record key against the row it reads before scoring it.
+
+Consequences:
+
+- One block record replaces \~1–4K per-vector records, reducing MDT record
count, write
+ amplification, compaction cost, and cleaner load by roughly three orders of
magnitude.
+- Scanning a cluster is one contiguous range read per segment — one per file
group the cluster
+ occupies, and exactly one for a single-segment cluster — rather than
thousands of point
+ lookups.
+- The array layout lets pass 1 decode only sign planes and factors, survivors
only the extra
+ planes, and finalists only the locators and keys (§6.2). Since the block is
fetched as one
+ record, this bounds scan and decode work rather than bytes read from storage.
+
+### 4.2 Row families
+
+The `vector_index_<name>` partition holds several record families under one
binary-sorted key
+scheme, so one prefix scan of a cluster returns its blocks and any fresh
deltas together:
+
+Keys are binary and begin with a one-byte family tag, so the table below is
directly checkable
+against `VectorIndexMetadataKey`:
+
+| Key family | Tag |
Cardinality | Purpose
|
+| ------------------------------------------------------------- | ------ |
-------------------------- |
---------------------------------------------------------------------------------------------------------------------
|
+| `__manifest__` | `0x00` | 1
| Active generation pointer and persisted-format
version. |
+| `M\|<generation>` | `0x01` |
generations | Generation state, vector schema, bootstrap
baseline, verified-contiguous frontier, and chunk counts/checksums. |
+| `Q\|<generation>\|<chunk>` | `0x02` |
chunks per generation | Size-bounded chunks of serialized quantizer state.
|
+| `T\|<generation>\|<chunk>` | `0x03` |
chunks per generation | Size-bounded chunks of the serialized `K × D`
centroid matrix. |
+| `C\|<generation>\|<cluster>` | `0x04` | K
per generation | Cluster manifest: routing version, segment count,
vector count, candidate file groups, and counters. |
+| `F\|<generation>\|<completionTime>` | `0x05` |
data writes after baseline | Atomic proof that the generation incorporated one
source data-write instant, keyed by that instant's completion time. |
+| `P\|<generation>\|<cluster>\|<segment>\|<blockId>` | `0x10` |
blocks per generation | **Posting block** (packed codes, factors,
locators, dictionaries, keys). |
+| `P\|<generation>\|<cluster>\|<segment>\|<DELTA>\|<recordKey>` | `0x10` |
deltas | Small per-record delta records appended between
compactions. |
+
+### 4.3 Posting segments
+
+Postings are placed by a two-level split. The IVF cluster is the first level.
The second is the
+**posting segment**, and **one posting segment is exactly one MDT file group**:
+
+```text
+IVF cluster ──split by size──> N posting segments ──1:1──> N MDT file groups
+```
+
+A cluster's `segmentCount` is derived from its vector count, with a minimum of
one, and is sized
+so that one segment fills a reasonable MDT base file. A small cluster
therefore occupies a single
+file group; a large one spreads across several. A cluster's postings live only
in that cluster's
+own file groups, so a hot cluster cannot become one oversized prefix range and
cannot crowd
+another cluster's file groups.
+
+Two naming choices matter here. The unit is a *segment* rather than a *shard*
so it does not
+collide with RLI or secondary-index sharding; a posting segment is not an RLI
shard, a data-table
+bucket, or a Spark partition. And its physical home is called a *file group*
rather than a shard
+because vector-index placement is explicit and size-derived, not
consistent-hashed the way RLI
+and SI placement is.
+
+The cluster manifest stores `segmentCount` and `routingVersion`. A writer
places a record within
+its cluster by `segmentId = hash(record_key) % segmentCount`, which keeps
segments balanced as
+new vectors arrive without requiring a key-range decision per insert. Changing
`segmentCount`
+changes every key's mapping, so maintenance must rewrite the whole cluster
under a new routing
+version and write the manifest change in the same MDT commit as that rewrite.
Large remaps use a
+new generation rather than mixing routing versions in one cluster range.
+
+Within a segment, entries are packed into blocks identified by `blockId`: a
counter local to one
+`(generation, cluster, segment)`, assigned at pack time in the sorted order of
§5.1 as each block
+fills. Ids are never global and never reused within a segment, so a repack
under Tier-1
+compaction (§7.5) reassigns them for that segment alone. Delta records carry
the reserved `DELTA`
+marker in place of a `blockId` and are not packed.
+
+Range-partitioning the intra-cluster key space — so that a segment holds a
contiguous run of the
+bootstrap `(fileGroup, rowPosition)` ordering rather than a hash-scattered
subset of it — is a
+follow-up optimization for a future index layout, not part of this design.
+
+### 4.4 Delta records
+
+Between compactions, per-commit vector writes append small **delta records**
at the end of the
+same cluster key range. A delta key is the posting prefix with the reserved
`DELTA` block id
+(`0xFFFFFFFF`) in place of a `blockId`, followed by the record key, so one
delta addresses exactly
+one record and repeated writes to the same record collapse on that key.
Because deltas share the
+cluster prefix with the packed blocks, a single prefix scan sees both in one
pass (§6, §7).
+
+Deltas are the mechanism by which updates and deletes reach the index. They
are written into the
+vector-index MDT file groups themselves, so a query runs its ordinary
quantized search over the
+segment's ranges and then applies the deltas as adjustments to what that scan
returned, before
+any exact value is fetched from the data table. Deltas are **record-scale**:
one entry per
+changed record, in contrast to the packed blocks they override, which is what
keeps a delta cheap
+to write and what makes periodic repacking (§7.5) necessary rather than
optional.
+
+### 4.5 Generation model
+
+A generation is a consistent set of centroid, quantizer, cluster, and
posting-block metadata:
+
+```text
+__manifest__ -> active generation id + persisted-format version
+M|<gen> -> generation metadata + schema + verified frontier + chunk
integrity
+Q|<gen>|... -> size-bounded quantizer-state chunks
+T|<gen>|... -> size-bounded centroid chunks
+C|<gen>|... -> cluster manifests
+P|<gen>|... -> posting blocks + deltas
+F|<gen>|... -> incorporated data-write markers
+```
+
+The builder allocates a generation id once and writes deterministic keys, so
retrying a
+partially completed build overwrites the same records. A `BUILDING` generation
is invisible
+to readers; bootstrap validates expected chunks, checksums, cluster/block
counts, schema, and
+memory budgets before one MDT commit changes it to `ACTIVE` and flips
`__manifest__`.
+Abandoned or invalid `BUILDING` generations are never activated and may be
garbage-collected.
+Superseded generations are marked `RETIRED` and kept until no retained
snapshot and no in-flight
+query still reads them. Generation is the only version axis: any centroid,
quantizer, routing, or
+posting change that cannot be made in place within one MDT commit creates a
new generation.
+
+---
+
+## 5. Bootstrap and Write Path
+
+
+
+### 5.1 Spark bootstrap
+
+Bootstrap and full rebuild are Spark-only in v1. They build a `BUILDING`
generation from a single
+table snapshot:
+
+```text
+1. Read latest file slices; extract key, partition, file group, base instant,
row position,
+ and vector bytes; train IVF centroids from a bounded sample.
+2. Validate K × D × elementWidth and driver/executor memory budgets before
materialization.
+3. Broadcast centroids; assign each vector, encode RaBitQ metadata, sort by
+ (cluster, fileGroup, rowPosition), and pack posting blocks.
+4. Write M|, size-bounded Q| quantizer and T| centroid chunks, C|, and P|
records under
+ deterministic keys.
+5. Validate chunk counts/checksums and posting counts, then activate the
generation.
+```
+
+The training sample satisfies percentage and per-cluster floors, for example
+`min(N, max(1M, 256*K, min(10M, 0.5%–1% of N)))`. Chunking keeps individual
MDT records
+bounded; readers reconstruct the matrix only after validating manifest
integrity metadata.
+
+### 5.2 Incremental inserts and vector updates
+
+The vector indexer follows the existing secondary-index and RLI update model.
Its vector-changing
+path groups the commit's `HoodieWriteStat`s by touched file group, reads the
current state from
+the commit's own bounded files, and reconstructs the preceding merged file
slice. It then
+compares rows by record key and appends deterministic posting deltas. This
path reads
+approximately twice the data of the touched file groups per commit: once for
current state and
+once for previous state. The amplification is bounded by the commit's touched
file groups and
+is the same kind of cost paid by secondary-index maintenance; it is not the
current-files-only
+expression-index pattern. A future relocation-only optimization may bypass
those reads only when
+the write path's existing committed outputs establish both that the canonical
vector bytes are
+unchanged and what the new locator is; without that, it takes the normal
reconstruction path.
+
+Classification is conservative and ordered. Vector equality is bytewise
equality of the
+canonical stored vector representation and may be claimed only when the
existing committed-
+output, payload, and precombine inputs provide both the authoritative old and
winning new
+bytes. If equality cannot be proven—including current-only catch-up after the
preceding slice
+was cleaned—the update is treated as vector-changing and is routed and encoded
again. This
+may do extra work but cannot preserve a stale cluster assignment by guessing.
+
+If vector bytes differ, the indexer tombstones the old cluster/segment posting
and writes a newly
+assigned and encoded posting. If bytes are proven equal but the locator
+(partition, file group, base instant, or row position) differs, the indexer
emits a locator
+refresh. An implementation may skip routing/re-encoding only when its existing
inputs retain
+the prior encoded posting and prove byte equality; otherwise it safely
recomputes. If both
+bytes and locator are equal, it emits no posting update. In particular,
rewriting an otherwise identical record
+under a new base instant is a locator refresh, not a no-op. An unchanged
vector is never
+reassigned merely because a later LIRE operation changed centroids;
compaction/LIRE owns that
+reassignment.
+
+The indexer preserves RLI locator semantics, including `rowPosition = -1` for
MoR log rows, and
+uses the existing key reduction rule to prefer a live update when a record
moves across file
+groups or partitions. It computes vector-specific cluster/segment assignment
and RaBitQ encoding
+against the ACTIVE generation, and emits posting deltas, additive cluster
statistics,
+and the commit's freshness marker in the same MDT commit. This uses the
existing per-data-
+commit `Indexer.buildUpdate` dispatch; `VectorIndexer` returns an `F|...`
record even without
+posting deltas, so its update is never empty and requires no new call site.
+
+### 5.3 Non-vector updates and deletes
+
+A non-vector update with an unchanged locator emits no posting update. A
locator change emits
+a refreshed posting even when the vector bytes are unchanged. When the
preceding slice is
+available, deletes emit old-posting tombstones and vector changes emit a
tombstone plus the new
+posting. The query-time RLI lookup is authoritative and suppresses stale or
deleted
+candidates independently of those tombstones (§6.3); posting tombstones
improve index hygiene
+and candidate efficiency but are not the sole correctness mechanism.
+
+Inline commit processing can rely on the preceding slice because it runs
inside the data
+commit and cleaning lags that commit. Catch-up replay may run after cleaning
removed the
+preceding slice. In that case replay explicitly degrades to current-only
reconstruction: it
+emits inserts and updates visible in the current state, but cannot infer
deletes or old posting
+keys. The freshness frontier and the RLI lookup then provide correctness until
maintenance removes
+the stale postings. This layering is required for legacy-writer gaps and
archival catch-up.
+The indexer still emits one freshness marker when a data write produces no
vector records,
+including schema-only or allowed-empty commits, so an empty update cannot
create a permanent
+false-stale gap.
+
+### 5.4 Engine support
+
+Snapshot extraction and centroid training are engine-specific; routing,
encoding, record
+construction, freshness markers, and committing them are common code. Spark
supplies
+bootstrap/rebuild in v1. Spark, Flink, and Java writers that know about the
vector partition can
+maintain an active generation through the common metadata-writer path. A
writer that does not
+know about it leaves the partition untouched, which is standard MDT behavior
for a partition a
+writer does not handle; its commits are then detected by the query-time
freshness gate (§9).
+
+### 5.5 Multi-writer rebuild and catch-up
+
+Rebuild and catch-up must work within Hudi's existing asynchronous indexing
protocol rather than
+running a parallel lifecycle beside it; §7.7 states that mapping.
+
+Rebuilds are replay-only: writers continue updating the active generation and
never dual-write
+to a `BUILDING` generation. After constructing at baseline `T_boot`, the
builder replays
+completed data writes in timeline order outside the coordination lock. Replay
uses the active
+timeline and, when repairing a gap older than its retained window, the
existing archived-
+timeline APIs. Each replay writes the new generation's deltas and
source-instant marker
+atomically. Replay is at least once per source instant. A committed marker
makes that instant
+at most once observed; if a process fails before the marker commit,
deterministic record keys,
+encoding, and key-level precombine make full re-execution harmless. When a
cleaned preceding
+slice is unavailable, replay uses the current-only degradation from §5.3
rather than claiming
+to reconstruct old posting keys. When the gap is bounded,
+`TransactionManager` and the metadata-index lock select `T_cut`, perform a
final micro-catch-up,
+verify contiguous coverage, and commit the new generation. If replay cannot
converge within its
+round/time budget, the attempt exits without activating anything; it neither
holds the lock
+indefinitely nor leaves partial state visible.
+
+---
+
+## 6. Read Path
+
+
+
+### 6.1 Query planning
+
+```text
+1. Resolve the table snapshot and validate its vector schema against the
active manifest.
+2. Check the generation's marker frontier against that snapshot's data-write
instant (§9).
+3. Load and validate generation metadata, centroid chunks, quantizer, and
cluster manifests.
+4. Probe centroids, select top-nprobe clusters, and resolve segment prefixes
and file groups.
+```
+
+### 6.2 Two-pass scan over posting blocks
+
+For each selected `(cluster, segment)`, prefix-scan
`P|<gen>|<cluster>|<segment>|*` (blocks +
+deltas) and score column-wise:
+
+```text
+Pass 1 (bound + prune): read S1 sign planes + S3 factors. For every vector
compute an
+ optimistic bound — the best distance it could possibly achieve. Skip
vectors whose
+ best case cannot beat the current K-th candidate (typically 85–95%
pruned).
+Pass 2 (refine): for survivors, read S2 extra bit planes and compute
the full
+ multibit unbiased estimate; keep the refineFactor·K best.
+Finalize: only the surviving finalists touch S4 locators and S6
keys to learn
+ where they live and who they are.
+```
+
+Scan cost scales with the number of vectors surviving each pass, not with
table size. Delta
+records encountered in the same scan are scored identically and supersede
matching packed
+entries.
+
+### 6.3 Freshness check and exact re-rank
+
+The Record Level Index (RLI) is a hard prerequisite. Vector-index creation
rejects a table
+without RLI, and the vector planner resolves the dependency again for every
query. If RLI was
+subsequently dropped by a path unaware of vector indexes, planning fails with
the missing
+dependency and remediation rather than serving results. RFC-109 does not
change the common
+index-drop path. Posting locators, including `rowPosition`, are hints;
finalist keys are
+validated by batched RLI lookup at the query's snapshot. Moved keys are
re-resolved and deleted
+keys are removed.
+
+Candidate generation retains a surplus before this check. If stale or deleted
candidates leave
+fewer than K live rows, the planner draws further candidates from the
retained, distance-ordered
+pool and repeats the RLI lookup, until K live rows are available or
`maxRerankCandidates` or the
+request deadline is exhausted. Exact mode then reads and scores the
authoritative vector at that
+snapshot. A COW/base-resident candidate may use positional Parquet fetch only
after resolving its
+live file slice at that snapshot. The decoded `_hoodie_record_key` must match
the finalist
+key. A key mismatch, changed base instant, stale decision, or invalid position
triggers key-
+based relocation in that same snapshot-resolved file slice; failure to locate
and validate the
+key invokes configured fallback or fails rather than scoring a different row.
+
+An MoR finalist with `rowPosition = -1` is discoverable through its delta
posting, validated
+through RLI, and fetched by key from the merged file slice resolved at that
snapshot. The
+fetch applies base-plus-log ordering, payload/precombine merge, and tombstone
semantics and
+must never substitute an older base-file version. Compaction can move that
record onto the
+faster positional path, but is not required for discoverability or exact
materialization.
+`indexLagInstants`, when present, is marker-frontier lag as defined in §9.
+
+RLI's schema already carries a position field. Populating it, or storing the
page index in RLI,
+would make exact fetch an O(1) read of the containing base-file page,
addressing the page directly
+rather than opening the file and scanning `_hoodie_record_key` to validate or
relocate. That is
+tracked as separate work; this design does not depend on it, and validates the
decoded key on
+every positional fetch either way.
+
+### 6.4 Batched queries
+
+For a relation of query vectors, the plan shares MDT work: load generation
metadata once,
+encode all queries, probe per query, group by selected `(cluster, segment)`,
scan each range
+once while maintaining a per-query top-R heap, then read the union of
candidate rows and
+exact re-rank per query.
+
+### 6.5 Fallback
+
+Index state that is missing, incompatible, stale, or out of budget does not
yield a silently
+incorrect answer. Depending on the configured policy, Hudi fails, warns with
`indexLagInstants`,
+or bypasses the index for an exact table scan. Exact-rerank defaults to `FAIL`
for stale state;
+callers that require strict availability may configure exact fallback.
+
+---
+
+## 7. Maintenance, Rebalancing, and Cleaner
+
+Table services are Hudi's automated maintenance runtime; the query path
remains correct independently of
+whether those services have run. For vector search indexes, correctness means
immediate candidate discoverability,
+no stale-version substitution, no silently short result, and exact
materialization of MoR
+merged snapshot through the merged file slice at the query's snapshot.
Maintenance reduces the
+cost of the expensive cases; it is not a dependency of query-path correctness.
+
+### 7.1 Correctness layer: immediate visibility
+
+Every completed vector update is immediately discoverable at its committed
snapshot before
+compaction, clustering, or rebuilding of index:
+
+1. The data file write emits a delta posting and source-instant markers to be
written to MDT vector search index atomically.
+2. Candidate generation overlays the delta on packed postings.
+3. RLI confirms the finalist's live logical location and prevents incorrect
reads from
+ an older physical version.
+4. Exact mode materializes base-resident finalists positionally with mandatory
key validation,
+ and resolves MoR log-resident finalists by key through the merged file
slice (§6.3).
+5. Exhaustion falls back or fails; it never silently returns an incomplete
result.
+
+Compaction may move log-resident candidates onto the faster positional path,
but requiring
+compaction for discoverability or permitting an old base version would provide
read-optimized
+rather than snapshot semantics and is forbidden. MDT is a MoR table as such
and this expectation
+is consistent with how MDT partitions are read today.
+
+### 7.2 Continuous write maintenance
+
+The existing MDT indexer dispatch is extended with a vector indexer. On each
data-table commit it
+appends posting deltas for inserts, vector updates, and deletes, updates the
per-cluster
+counters, and writes that commit's `F|` marker, in the MDT commit that data
commit already makes.
+
+Clustering and compaction rewrite files without changing any vector value, but
they relocate rows
+differently. Clustering issues a `replacecommit` that rewrites records into
new file groups.
+Compaction rewrites one file group's base file at a new instant, leaving the
file group in place
+but changing row positions within it. Record keys survive both.
+
+Because a posting stores a row locator — file group, instant, and row position
(§4.1) —
+either rewrite invalidates the locators of the rows it moved, so the vector
index must refresh
+them. This is the same class of maintenance the record-level index already
performs on such a
+commit; the secondary index is unaffected because it maps to record keys,
which do not change.
+
+What neither rewrite requires is re-routing or re-encoding, since the vector
bytes are unchanged.
+The baseline path may still reconstruct and re-encode the touched vector. A
relocation-only branch
+may skip routing and encoding, but only where the write path's existing
committed outputs both
+establish byte equality and retain the prior encoded posting. Skipping that
work is therefore an
+optimization, not something correctness relies on.
+
+### 7.3 Demand-aware data-table compaction
+
+Data-table compaction converts MoR log-resident rows into base rows with
stable positions.
+The posting records are updated, so they are absorbed back to the positional
exact-fetch
+path on the base file. Merged-slice fetch preserves correctness before
compaction;
+compaction only bounds the population requiring that more expensive read.
+
+In the first version, we will work with the built-in MDT compaction strategy
that picks
+candidate index segments to compact based on size of accumulated deltas.
+However, longer term, a vector-aware `CompactionStrategy` may prioritize file
groups that repeatedly contain query
+finalists. The query path emits metrics tagged by table, partition, and file
group, including
+`logResidentFinalistCount`. This requires telemetry collection on query
patterns perhaps in another system-wide
+Hudi table. An external component may aggregate and decay those metrics; the
proposed compaction strategy then reads
+those statistics and combines demand with structural cost, for example:
+
+```text
+priority = decayedDemand * logResidentRatio * logBytes /
estimatedCompactionCost
+```
+
+The strategy must tolerate absent, stale, or partial demand data. Without it,
selection falls
+back to structural inputs such as log-resident ratio, log bytes, and commit
count, preserving
+safe stock-like behavior. The same metrics-only rule applies to future signals
such as
+`staleCrowdingRate` and `fetchKeyMismatch`.
+
+### 7.4 Vector-affinity clustering
+
+Data-table clustering may improve exact-fetch performance by ordering rows
using active-generation
+vector affinity. Cluster assignment is derived state scoped to a generation,
so the planner
+resolves one ACTIVE generation for routing and records its id in the
clustering plan. An
+execution whose generation precondition no longer holds is replanned or
aborted; it never
+mixes assignments from two generations.
+
+This is an optional optimization and requires benchmark evidence. At minimum
the benchmark
+compares:
+
+- business-key-primary, with `clusterId` and `segmentId` as secondary keys;
+- `clusterId`/`segmentId` primary, which maximizes vector locality while
pricing the loss of
+ business-key locality; and
+- the unclustered baseline.
+
+It measures candidate file groups, rows per selected page, ranged bytes,
exact-fetch latency,
+and locator-update volume. Re-encoding unchanged vectors is measured
separately so a future
+proven relocation-only optimization can price its benefit; correctness does
not assume it.
+Assignment drift after LIRE or generation replacement makes this layout
periodically renewable
+rather than a correctness invariant.
+
+### 7.5 Tier-1 vector MDT compaction
+
+`deltaCount` and `tombstoneCount` are structural ClusterStats inputs for
scheduling semantic
+compaction of the vector MDT partition. This service does more than change a
trigger: it folds
+deltas into deterministically sorted packed blocks, applies delta precedence,
RLI-drops dead
+entries, consolidates current locators, resets counters. This bounds
posting-scan cost independently
+of data-table compaction, which instead bounds exact-fetch cost.
+
+### 7.6 LIRE split/merge maintenance (future extension)
+
+Note: to be revisited after shipping v1 with built-in Hudi compaction
strategies and evaluating performance
+systematically.
+
+Sustained `liveCount` skew may eventually schedule a local split or merge.
**LIRE is not part
+of the first implementation.** Until it lands, skew and centroid drift use
full generation
+rebuild and activation. LIRE also lands as a new generation rather than a
second routing version
+inside the current one. What follows is the requirement for a later LIRE
implementation, not
+present behavior.
+
+The service reads the affected clusters' authoritative vectors, runs local
reassignment,
+writes routing/centroid metadata, posting ranges, and ClusterStats into a
BUILDING generation,
+and activates that generation after validation. All of a rebalance's changes
therefore become
+visible in one commit: a query that resolved the generation before activation
sees the old one for
+its whole life, and one resolving at or after activation sees the complete new
one. There is no
+independently visible centroid, routing, posting, or manifest step in between.
+
+LIRE follows Hudi clustering's schedule/execute/rollback pattern, though MDT
record replacement is
+not a data-table `replacecommit`. Two properties remain to be shown by test:
+
+- **Cache coherence.** A query that cached routing or centroids before
activation continues to use
+ that generation for its lifetime, so a cache never combines old routing with
new postings.
+- **Retention.** The cleaner keeps the old MDT files for at least the duration
of the
+ longest-running query, and does not retire them while a supported snapshot
can reference them.
+
+### 7.7 Catch-up and generation rebuild
+
+Catch-up/gap repair and full generation rebuild reuse and extend the
asynchronous indexer
+lifecycle; they are not assumed to be fully hosted by it in the initial
implementation. The
+intended lifecycle is: schedule; allocate a `max + 1` BUILDING generation
under the lock; build
+from a baseline snapshot; replay completed source instants; run a bounded
final catch-up under the
+metadata lock; verify the frontier is contiguous through `T_cut`; activate;
retire the preceding
+generation later. Gap repair is a bounded invocation of the same replay path.
Neither operation
+leaves partial state visible, nor requires writers to write to two generations.
+
+### 7.8 Triggers and cleaner coordination
+
+| Trigger | Signal
| Action
|
+| ------------------------------------- |
--------------------------------------------------------- |
---------------------------------------------------------------------------- |
+| MoR log-resident exact-fetch pressure | structural log metrics plus optional
decayed query demand | Prioritize data-table file groups for compaction.
|
+| Posting delta accumulation | `deltaCount` / `tombstoneCount`
| Semantically compact the vector cluster range.
|
+| Cluster imbalance | `liveCount` over split/merge
thresholds | V1: rebuild a full generation; future: schedule
one LIRE generation. |
+| Centroid drift | bounded drift statistic over
threshold | Build and activate a new generation.
|
+| Large physical rewrite | replaced file groups
| Refresh locators; skip re-encoding only where byte
equality is established. |
+| Frontier gap | missing source-instant marker
| Run bounded catch-up replay.
|
+| Periodic validation | scheduled health check
| Compare postings against authoritative table state.
|
+
+The cleaner retains vector generations and pre-LIRE files for as long as any
supported snapshot
+may need them. Visible entries must resolve to live base-table records for
every
+snapshot the vector index serves.
+
+---
+
+## 8. Spark API Surface
+
+Vector search should eventually be expressed in the query's own filter
predicates, through
+dedicated syntax or an expression index, so that a result need not be joined
back against the
+source table. The first implementation uses a table-valued function for
simplicity.
+
+Table-valued function (single-query):
+
+```sql
+SELECT * FROM hudi_vector_search(
+ table => 'products',
+ column => 'embedding',
+ query => ARRAY(0.1, 0.2, ...),
+ k => 10,
+ metric => 'cosine', -- l2 | cosine | dot_product
+ mode => 'approximate' -- or 'exact_rerank'
+);
+```
+
+Datasource options (integration/testing):
+
+```javascript
+hoodie.datasource.read.vector.index.name=embedding_idx
+hoodie.datasource.read.vector.query.vector=[0.1,0.2,...]
+hoodie.datasource.read.vector.query.nprobes=32
+hoodie.datasource.read.vector.query.topk=100
+```
+
+Index DDL:
+
+```sql
+CREATE INDEX embedding_idx
+ON products
+USING VECTOR (embedding)
+OPTIONS (
+ 'vector.metric' = 'cosine',
+ 'vector.quantizer' = 'IVF_RABITQ',
+ 'vector.num_clusters' = '4096',
+ 'vector.rabitq.bits' = '4',
+ 'vector.rabitq.seed' = '42',
+ 'vector.rabitq.assume_normalized' = 'false',
+ 'vector.query.nprobes' = '32',
+ 'vector.query.refine_factor' = '50',
+ 'vector.query.stale_policy' = 'FAIL' -- FAIL | WARN | FALLBACK
+);
+```
+
+V1 validates the complete option map, rejects unknown or retired keys, and
persists canonical
+values with explicit defaults:
+
+| Option | Default | Constraint
|
+| --------------------------------- | -------------- |
------------------------------------------- |
+| `vector.metric` | `cosine` | `cosine`, `l2`, or
`dot_product` |
+| `vector.quantizer` | `IVF_RABITQ` | `IVF_RABITQ` in v1
|
+| `vector.num_clusters` | `256` | positive integer
|
+| `vector.max_iter` | `20` | positive integer
|
+| `vector.rabitq.bits` | `4` | integer from 1 through
8 |
+| `vector.rabitq.seed` | `42` | signed 64-bit integer
|
+| `vector.rabitq.assume_normalized` | `false` | boolean
|
+| `vector.query.nprobes` | `32` | positive and no greater
than `num_clusters` |
+| `vector.query.refine_factor` | `50` | positive integer
|
+| `vector.query.mode` | `exact_rerank` | `approximate` or
`exact_rerank` |
+| `vector.query.stale_policy` | `FAIL` | `FAIL`, `WARN`, or
`FALLBACK` |
+
+Vector dimension and element type are intentionally absent: they come only
from the indexed
+column's `VECTOR(D[, elementType])` schema.
+
+---
+
+## 9. Correctness, Compatibility, and Freshness
+
+- **Source of truth.** The base-table `VECTOR` column and its
dimension/element type are
+ authoritative; MDT is an acceleration structure. Exact reranking reads
base-table values.
+- **Generation consistency.** One generation supplies the format, schema,
centroids,
+ quantizer, routing, postings, and markers for a query. Readers reject
unsupported or mixed
+ generation formats rather than partially interpreting them.
+- **Snapshot.** MDT reads, RLI lookups, and base-table reads within a query
all use the snapshot
+ resolved at planning.
+- **Visibility.** At most one live entry per record reaches the result,
through delta precedence,
+ the mandatory RLI lookup, and single-commit generation activation. Posting
tombstones are an
+ optimization and a maintenance signal, not a substitute for the RLI lookup.
+- **Ranking.** RaBitQ selects candidates; exact mode orders finalists by
base-table distance.
+
+### 9.1 Compatibility with unaware writers, and the marker frontier
+
+Readers and writers that do not know about a vector partition ignore it, which
is standard MDT
+behavior; a **data-write instant** is a completed `commit`, `deltacommit`, or
`replacecommit`
+on the data-table write timeline, including allowed empty or schema-only
instances of those
+actions, while table-service actions such as clean are excluded.
+
+For every data-write instant after a generation's bootstrap baseline, the
vector indexer
+writes `F|generation|completionTime` — keyed by the data-write instant's
completion time, not its
+requested time — in the same MDT commit as that instant's vector delta records;
+when there are no vector deltas, the marker is the non-empty vector-index
update. The index is
+fresh through `F` only if every completed data-write instant from the baseline
through `F` has
+a marker and no earlier data-write instant remains inflight. The generation
manifest persists
+a verified-contiguous frontier, never the maximum marker observed. Ordinary
advancement and
+planning use that checkpoint plus the active timeline only. The frontier
advances only after
+contiguous marker coverage is verified; its manifest update is committed
atomically with the
+final marker/deltas that close the gap, or in a later MDT commit, never before
that evidence.
+If the checkpoint predates the earliest retained active instant, the uncovered
interval is
+stale/unknown rather than implicitly skipped, so archiving cannot turn a
missing legacy-writer
+marker into false freshness. Catch-up alone reads the archived timeline
through existing
+public APIs to repair the interval, write its missing deltas/markers, and
advance the
+checkpoint. A frontier behind the query snapshot's data-write instant invokes
`FAIL`, `WARN`, or
+exact `FALLBACK`.
+
+A rollback needs no marker mutation rule. Ordinary planning uses the active
timeline, while
+catch-up reconstructs the effective completed data-write history from active
and archived
+timeline APIs; rolled-back writes do not require coverage. Thus an unaware or
legacy writer
+cannot falsely certify freshness; it leaves an observable, repairable gap.
+
+### 9.2 OCC and NBCC concurrency
+
+These are the required semantics; §10 makes each race an acceptance test. They
are not a claim
+that every row already has scale evidence.
+
+| Scenario | Serialization / winner
| Frontier and reader result
|
Retry / rollback
|
+| --------------------------------------------------- |
------------------------------------------------------------------------------------------------------------------------------------------------
|
------------------------------------------------------------------------------------------------------------
|
----------------------------------------------------------------------------------------------------
|
+| Two vector-aware OCC writers | Existing OCC conflict
resolution must choose the committed data-table winner.
| Each completed winner must have
one atomic delta/marker; readers must use the query's timeline and RLI. |
Loser must retry from refreshed table state; no marker for an aborted instant
is required. |
+| OCC writer concurrent with generation rebuild | Writers must update
only ACTIVE; rebuild must replay from its baseline and choose a locked cut.
| BUILDING must stay invisible
until contiguous through the cut, then activate atomically.
| Failure or non-convergence must abandon BUILDING without changing ACTIVE.
|
+| Out-of-order NBCC deltacommits | Timeline completion
order may differ from requested-time order; markers are keyed by completion
time and the frontier is verified in that order. | A later marker must not
advance the verified frontier across an earlier inflight or missing instant.
| Completion or catch-up must close the gap; rollback removes the instant
from effective history. |
+| Vector-aware and legacy/unaware writers | Both must use normal
data-table conflict semantics; only the aware writer emits vector metadata.
| The missing legacy marker must
make the frontier observably stale; policy fails, warns, or exact-falls back. |
Catch-up must use existing active/archive timeline and RLI semantics; it must
never invent coverage. |
+| Rebuild concurrent with clustering/replacecommit | The replacecommit must
be treated as a source data-write instant and replayed like other writes.
| Activation must require its marker
and relocated live keys through the cut. |
Rebuild must retry catch-up or abort; readers remain on the old ACTIVE
generation. |
+| Async MDT compaction concurrent with vector indexer | Existing MDT
commit/lock semantics must serialize metadata updates.
| Compaction must not
advance a frontier beyond committed marker evidence.
| Standard metadata retry/rollback must replay deterministic keys;
partial output remains invisible. |
+
+---
+
+## 10. Test Plan
+
+- Unit: vector schema validation and byte conversion; MDT key generation and
prefix ordering;
+ RaBitQ deterministic encoding, query transform, multibit plane scoring,
factor handling,
+ and the pruning bound (approximate estimate within its stated error of
exact).
+- Unit: posting-block round-trip, delta supersession, segment/routing-version
stability,
+ marker-frontier contiguity (including empty updates, inflight gaps, and
archival rollover),
+ candidate refill, and two-pass pruning correctness.
+- Bootstrap/recovery: deterministic retry after partial writes; centroid chunk
count/checksum;
+ memory-budget rejection; validation before activation; safe
abandoned-generation cleanup.
+- Read path: metadata cache loading, centroid probing, block prefix scan,
two-pass scoring,
+ RLI lookup, positional exact fetch, top-K reduction, bounded continuation,
and the legacy-gap
+ case where catch-up runs after the preceding slice has been cleaned. COW
fixtures
+ rewrite row positions through compaction/clustering and require decoded-key
validation plus
+ key-based relocation. MoR fixtures create log-resident finalists and verify
exact materialization
+ from the merged file slice before compaction, with no stale base-version
substitution;
+ compaction only moves those rows onto the positional fast path.
+- Incremental maintenance: COW and MoR insert, bytewise vector update,
cluster-changing
+ update, locator-only rewrite, delete, empty commit, deterministic replay
after failure before
+ the marker commits, and file-group/partition move reduction. MoR coverage
verifies
+ `rowPosition = -1` for log rows. Unchanged-vector coverage proves byte
equality from existing
+ committed-output/payload inputs; unavailable equality takes the conservative
vector-changing
+ branch.
+- Maintenance services: demand-aware compaction degrades safely when telemetry
is absent;
+ query execution emits metrics but performs no MDT writes; Tier-1 compaction
applies
+ precedence/liveness, deterministic repacking, frontier checkpointing, and
evidence-gated
+ marker pruning; affinity clustering benchmarks both business-key-primary and
+ cluster-primary layouts.
+- Future LIRE validation (not a first-implementation gate): one MDT commit
carries all affected
+ routing, centroid, posting, stats, and manifest changes; caches loaded
before activation remain
+ coherent; cleaner retention covers the longest-running query.
+- Concurrency: execute every OCC/NBCC scenario in §9.2, including out-of-order
completion,
+ rollback, rebuild cut-over, replacecommit relocation, and async MDT
compaction.
+- Freshness layering: simulate an unhooked legacy commit with no marker or
posting tombstones;
+ verify frontier gating, then catch up after cleaning the preceding slice and
verify deleted
+ records remain absent through the RLI lookup.
+- Datasource / SQL: vector read options, candidate file pruning, TVF
approximate and
+ exact-rerank modes.
+- Integration: ANN vs brute-force quality; upsert, delete, compaction,
clustering, rollback,
+ and restore; feature-aware and unaware writers; missing-RLI planner failure;
stale
+ `FAIL/WARN/FALLBACK`; catch-up and multi-writer generation activation;
cleaner retention.
+
+---
+
+## 11. Rollout and MVP Scope
+
+RFC-109 lands as a sequence of small, focused, generic PRs tracked by umbrella
issue
+[#19094](https://github.com/apache/hudi/issues/19094), so the community can
review
+architecture separately from code, the storage model separately from
execution, bootstrap
+separately from the read path, and lifecycle separately from scoring:
+
+| PR / child issue | Scope
|
+| --------------------------------------------------------------- |
----------------------------------------------------------------------- |
+| [PR #19309](https://github.com/apache/hudi/pull/19309) / #19095 | This RFC
document (docs-only). |
+| #19096 | Vector
index options and Spark DDL scaffolding (§8). |
+| #19097 | MDT row
families and posting-block payload schema, multibit-aware (§4). |
+| #19098 | RaBitQ
encoder/scorer contract, multibit-aware (§3). |
+| #19099 | Spark
bootstrap: IVF clustering + posting-block generation (§5). |
+| #19100 | Generation
visibility and reader consistency (§4.5, §9). |
+| #19101 | Query
planning via centroid probing (§6.1). |
+| #19102 |
Posting-block scan + approximate top-K candidate generation (§6.2). |
+| #19103 | Exact
rerank from authoritative base-table vectors (§6.3). |
+| #19104 | Drop,
rebuild, and cleanup semantics (§7). |
+| #19105 | Unit,
integration, and Spark SQL coverage (§10). |
+
+**MVP:** MDT-backed posting-block storage; Spark-first; IVF clustering and
probe planning;
+multibit RaBitQ approximate candidate generation; partition pruning before
candidate selection;
+exact rerank from base-table vectors; create / drop / rebuild semantics;
correctness and
+integration tests.
+
+**Non-goals for first landing:** ANN families beyond IVF + RaBitQ; arbitrary
predicate-plus-kNN
+planning over non-partition columns; time-travel index reads; non-Spark
bootstrap, rebuild, or
+query integration; GPU encoding; advanced auto-tuning; and LIRE local
split/merge maintenance.
+
+---
+
+## 12. References
+
+1. **RaBitQ: Quantizing High-Dimensional Vectors with a Theoretical Error
Bound for
+ Approximate Nearest Neighbor Search** — Gao & Long, SIGMOD 2024. Randomized
binary
+ quantization with an unbiased estimator and per-vector error bound.
+2. **Extended / multi-bit RaBitQ** — higher-bit residual quantization with the
same bound.
+3. **Product Quantization for Nearest Neighbor Search** — Jégou, Douze,
Schmid, IEEE TPAMI 2011.
+4. **Efficient and Robust Approximate Nearest Neighbor Search Using HNSW
Graphs** — Malkov,
+ Yashunin, IEEE TPAMI 2018.
+5. **SPFresh: Incremental In-Place Update for Billion-Scale Vector Search** —
local
+ incremental cluster maintenance and NPA invariants (LIRE inspiration).
+6. **Apache Hudi Metadata Table** — <https://hudi.apache.org/docs/metadata>
+
+---
+
+## Appendix A. Controlled BIGANN Research Evidence
+
+All rows use BIGANN 1B, 128-dimensional byte vectors, L2, `topK=10`, 100
ordered queries,
+and two passes per configuration. Recall is identical across the two passes;
latency below is
+the warm second pass. Exact means exact reranking of IVF candidates, not
exhaustive 1B search.
+
+### A.1 Current isolated acceptance run
+
+| Mode | `nprobe` | `refineFactor` | Recall\@10 | Warm mean / p50
/ p95 (s) |
+| ---------------- | -------- | -------------- | ---------- |
------------------------- |
+| approximate-only | 16 | not used | 0.822 | 7.561 / 7.485 /
8.639 |
+| approximate-only | 32 | not used | 0.857 | 7.586 / 7.383 /
8.740 |
+| approximate-only | 64 | not used | 0.871 | 7.762 / 7.596 /
9.508 |
+| exact-rerank | 32 | 50 | **0.960** | 11.947 / 11.869
/ 12.728 |
+
+This run ingested and indexed 1,000,000,000 rows in a fresh COW table using
16,384 clusters
+and 4-bit residual RaBitQ. The application requested 512 vector MDT file
groups, but post-run
+physical enumeration found 16,384 groups because that setting was not
propagated into the Hudi
+write configuration. The query measurements remain valid for the physically
realized 16,384-
+group layout; they are not evidence for a 512-group layout. A corrected
512-group rebuild is
+required before making that sizing claim. The run's immutable provenance is:
+
+- run id: `rfc109-1b-e1c94554-20260827`;
+- source commit: `e1c94554cd33cc49ee529f0a6e2b816f93045dbc`;
+- applied production patch SHA-256:
+ `c2c82d4d5648f89e0627e5cccbe08ebb4950f8bb3c7d535a9cbba6eaebdcbb6f`;
+- Spark bundle SHA-256:
+ `d6296ef2a66f6d1faf3523a858486b6cd5d9b25620650afaa25ce04739100bbc`;
+- corrected approximate harness SHA-256:
+ `bb5a691c99df44c9c164a3ee07d5a9b36fcf981880fa0dc637a6577fbe526817`;
+- corrected approximate result SHA-256:
+ `065772587e90f150a0102c8327b55d67cad94f0542766c333519753c826ba210`.
+
+The patch digest matters: these results include the committed-output RLI/SI
write-status fix
+that was not part of the named source commit. The first approximate artifact
from this run is
+excluded because its harness used retired option names and therefore executed
default exact
+mode; the table above uses only the corrected, hash-pinned artifact.
+
+### A.2 Historical controlled comparison
+
+The prior controlled run used the same query count, pass count, `topK`, and
exact
+`nprobe=32/refineFactor=50` protocol:
+
+| Mode | `nprobe` | `refineFactor` | Recall\@10 | Warm mean / p50
/ p95 (s) |
+| ---------------- | -------- | -------------- | ---------- |
------------------------- |
+| approximate-only | 16 | not used | 0.824 | 8.143 / 6.711 /
18.240 |
+| approximate-only | 32 | not used | 0.843 | 8.589 / 6.854 /
18.966 |
+| approximate-only | 64 | not used | 0.845 | 9.705 / 7.885 /
20.726 |
+| exact-rerank | 32 | 50 | **0.983** | 13.341 / 11.159
/ 24.162 |
+
+The comparison shows quality variability between independently trained
generations even with
+seed 42, while the current exact run improves warm mean by about 10.4% and p95
by about 47.3%
+at a 2.3-point recall cost. Smaller historical 20-query, one-pass parameter
sweeps are not
+used because they are not latency-comparable to this protocol.
