JunRuiLee opened a new pull request, #515: URL: https://github.com/apache/paimon-rust/pull/515
### Purpose Linked issue: part of #514 — PR 1 of 5 (does not close the issue) This is the first, read-only slice of primary-key vector (bucket-local ANN) search support. It mirrors Apache Paimon (Java) apache/paimon#8549 as of commit `a50a36ff8`. Java primary-key vector indexes append a nullable `_SOURCE_META` blob to `GlobalIndexMeta` listing the ordered source data files and row counts of a bucket-local ANN payload; an ANN ordinal maps to a `(data file, physical row position)` pair. This PR gives paimon-rust the ability to **parse** that metadata and **resolve** an ordinal — the foundation PR 2 (bucket search) and PR 3 (physical-position read) build on. Read/parse only. Out of scope for this PR (tracked in #514): ANN search, manifest selection, physical-row materialization, index building, and engine SQL entry points. ### Brief change log - Add a nullable `source_meta: Option<Vec<u8>>` field to `GlobalIndexMeta` (serde `_SOURCE_META`, `serde_bytes`), matching Java's 6th `GlobalIndexMeta` field. - New `spec/pk_vector_source.rs`: `PkVectorSourceFile` / `PkVectorSourceMeta`, a byte-for-byte parser for Java's `PkVectorSourceMeta` frame (version `1`, count, `writeUTF` filenames, `writeLong` row counts, trailing-bytes check), and an ordinal-only `resolve(ordinal) -> (file, position)` prefix-scan resolver. - Decode `_SOURCE_META` through the hand-written Avro index-manifest decoder, **schema-guarded**: a pre-#8549 5-field `_GLOBAL_INDEX` writer schema is detected from the file's writer schema and the 6th field is not read, so old manifests never misalign the byte stream. - The `_SOURCE_META` inner blob is Java `DataOutput` (big-endian ints/longs, modified UTF-8), parsed independently of the outer Avro layer; parsing helpers are module-private (no shared `common/` extraction in this PR). ### Tests All in `spec/pk_vector_source.rs` and `spec/index_manifest.rs` (22 tests; full `cargo test -p paimon` green): - Frame parsing: single/multi source file; rejects bad version, zero count, trailing bytes, truncated input, negative row count, empty list. - Java modified UTF-8: ASCII/2-byte/3-byte/supplementary round-trips; `C0 80` → NUL; raw `0x00` accepted (Java `readUTF` parity); truncated/bad-continuation and lone-surrogate rejected. - Resolver: single-file, multi-file prefix-sum boundaries, negative/out-of-range ordinal errors, `checked_add` overflow branch. - Avro: new-format `_SOURCE_META` round-trips (`Some`/`None`); **legacy 5-field `_GLOBAL_INDEX` decodes without misalignment and yields `source_meta = None`**. ### API and Format - No storage-format change originated here: this reads the `_SOURCE_META` format defined by Java apache/paimon#8549. `_SOURCE_META` is a nullable trailing field, so existing index manifests remain readable (verified by the legacy-schema test). - Public API additions only (`PkVectorSourceFile`, `PkVectorSourceMeta`); no existing API changed. ### Documentation No user-facing documentation changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
