chrevanthreddy opened a new pull request, #19317: URL: https://github.com/apache/hudi/pull/19317
### Describe the issue this Pull Request addresses Resolves #19097 (umbrella: #19094, RFC-109) Second foundational slice of RFC-109 (Hudi Native Vector Index): the Metadata Table (MDT) storage model and payload schema for vector indexes. Builds on the option surface in #19096. DDL command dispatch, bootstrap/write execution, and the read path land in dependent follow-up PRs (#19098, #19099, #19100…). ### Summary and Changelog - **Avro schema** (`HoodieMetadata.avsc`): add the `VectorIndexMetadata` union with all typed row families — `HoodieVectorIndexManifest`, `HoodieVectorIndexQuantizer`, `HoodieVectorIndexCentroids`, `HoodieVectorIndexPostingBlock`, `HoodieVectorIndexPostingDelta`, `HoodieVectorIndexClusterStats`, `HoodieVectorIndexTombstone`. Encoding metadata (RaBitQ bits, seed, residual/extended codes, scalars) is versioned per generation for multibit-aware reads. - **`HoodieMetadataPayload`**: new `vectorIndexMetadata` field + offset-based serde, field/entry-type constants, dedicated constructor, and factory builders for centroid/quantizer/manifest/cluster/posting records. - **Binary record keys**: `VectorIndexMetadataKey` (family-tagged binary key codec) + `VectorClusterRawKey`, `VectorGenerationManifestRawKey`, `VectorPostingPrefixRawKey` (`RawKey` impls). - **`HoodieTableMetadataUtil`**: binary key-family predicates/getters and cluster-aware file-group routing (`mapVectorPostingKeyToFileGroupIndex`). - **`MetadataPartitionType.VECTOR_INDEX`**: new partition type (ordinal 8) wired for availability, payload construction, partition path, and file-group mapping. - **`PostingBlockBuilder` / `PostingBlockView`**: packed posting-block encode/decode over the generated `HoodieVectorIndexPostingBlock`. - **Tests**: `TestVectorIndexMetadataPayload` (2) round-trip payload serde; `TestVectorIndexMetadataKey` (6) binary key encode/decode. Existing `HoodieMetadataRecord` positional callers updated for the new field. ### Impact Additive to the MDT schema (new optional union field, default `null`) and a new metadata partition type that is only populated via explicit vector `CREATE INDEX` (not enabled by any metadata config flag). No behavioral change to existing partitions, writers, or readers. ### Risk Level low Schema change is an optional field defaulting to `null` (backward/forward compatible); new partition type is inert unless a vector index is created. `hudi-common` compiles clean and vector serde + affected existing serde tests pass on current master. ### Documentation Update Row families and encoding-metadata semantics are documented as Avro `doc` fields and Javadoc. User-facing RFC document tracked in #19309. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Change Logs and Impact described above - [x] Necessary doc changes done or have another open PR - [x] For large changes, please consider breaking it into sub-tasks under an umbrella issue (#19094) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
