chrevanthreddy opened a new pull request, #19300: URL: https://github.com/apache/hudi/pull/19300
### Describe the issue this Pull Request addresses Adds the Metadata Table (MDT) storage foundation for native vector search proposed in RFC-104 (umbrella #19094): the Avro record types, payload wiring, partition type, and posting-block codec that later PRs (encoder, bootstrap, read path) build on. Closes #19097 ### Summary and Changelog Users gain no behavior change yet; this establishes the on-disk/serde contract for the vector index so subsequent PRs can encode, write, and read it. Changes (all in `hudi-common`, additive): - `HoodieMetadata.avsc`: seven vector index record types — centroids, quantizer, manifest, cluster stats, posting block, posting delta, tombstone. - `HoodieMetadataPayload`: new `VectorIndexMetadata` field, entry-type constants, and serde for the vector records (co-exists with existing record-index logic). - `MetadataPartitionType.VECTOR_INDEX`: new partition type plus record-type wiring. - `HoodieTableMetadataUtil`: posting-key → file-group mapping. - Posting block codec: `PostingBlockBuilder` / `PostingBlockView` (structure-of-arrays packing of ~1–4K vectors per record). - Vector index enums: `VectorIndexOptions`, `VectorIndexType`, `VectorDistanceMetric`. - MDT raw keys: `VectorIndexMetadataKey`, `VectorPostingPrefixRawKey`, `VectorClusterRawKey`, `VectorGenerationManifestRawKey`. - Tests: `TestVectorDistanceMetric`, `TestVectorIndexMetadataKey`, `TestVectorIndexMetadataPayload` (17 cases). Existing `HoodieMetadataRecord` all-args callers (`TestAvroRecordSizeEstimator`, `TestBufferedRecordSerializer`) updated for the appended field. ### Impact Adds one nullable field (`VectorIndexMetadata`) to the `HoodieMetadataRecord` Avro schema and a new `MetadataPartitionType`. The field is optional/defaulted, so existing MDT records and readers are unaffected; the new partition type is inert until a vector index is created (later PRs). No public API removed. No performance impact on existing partitions. ### Risk Level low Schema change is purely additive (new nullable field + new record types + new partition type), exercised by unit tests; existing metadata read/write paths are untouched apart from the appended constructor arg, which is covered by the updated tests. Full `hudi-common` compiles and tests pass on current master under JDK17. ### Documentation Update none (user-facing docs land with the feature-complete read/write PRs). The design is documented in RFC-104 (#19291). ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
