JunRuiLee opened a new pull request, #746: URL: https://github.com/apache/paimon-rust/pull/746
### Purpose A primary-key vector search has to be planned per bucket. Deciding which ANN segments are current needs the bucket's complete active-file set, which an ordinary table-scan split does not carry, so the planning object is a `BucketVectorSearchSplit` rather than a `DataSplit`. On the Java side that split could so far only be moved by Java object serialization, which keeps the search inside a JVM. apache/paimon#9386 gives it a byte form; this adds the reader for that form, so a Java planner can dispatch buckets to workers running this crate. That is the shape an engine integration needs: plan in the JVM frontend, search natively per bucket. This change decodes only. Planning stays in Java, and nothing here produces the format. Depends on apache/paimon#9386 — the fixture is shared, so the Java change has to land first. Part of #514. ### Tests `crates/paimon/src/table/pk_vector_bucket_split.rs` tests, 20 in total. The cross-language one decodes the message apache/paimon#9386's test generates, checked in verbatim as `goldens/bucket_vector_search_split_v1_datasplit_v9.bin`. Its split carries a non-empty partition row, non-empty min/max keys and non-empty stats on purpose: the nested `BinaryRow` layout is the part of the message a reader is most likely to get wrong, and an empty row exercises none of it. The rest are hostile input: bad magic, unsupported version, a byte-order marker for the other order, truncation in a fixed field and in a repetition, trailing bytes, a negative count, a count larger than the message could hold, an invalid presence flag, invalid UTF-8, a descending row range, a range past the end of its file, overlapping ranges, ranges naming a file the bucket does not hold, and duplicate data files. Plus the shapes the Java fixture cannot reach: absent optionals, a bucket with no payload and no ranges, and non-ASCII names. ### API and Format Notes - `BucketVectorPayload` is a distinct type rather than `spec::IndexFileMeta`, which models the manifest form: it has no external path and narrows the row count to `i32`, and both matter here. - Version 1 pins the envelope, not the bytes nested in it. The framed payload is `DataSplit::serialize` verbatim, so it follows that format's own version. The envelope's byte-order marker records which order the nested rows were written in, since they use the producer's native order; a message from the other order is rejected rather than misread. The marker does not record the nested version, so a future `DataSplit` version can appear inside an envelope still labelled version 1. - The golden is a copy, so a Java-side regeneration has to be carried over by hand. The comment on it names the source path and the test that writes it. ### Out of Scope Reviewing this surfaced existing behaviour of the `DataSplit` decoder that this change deliberately leaves alone, since it predates it and is reached from every split path, not just this one: an absent `totalBuckets` becomes `1` instead of staying absent, `isStreaming = true` is rejected, snapshot `-1` is rejected, a deletion-file list of a different length than the data files is rejected, and only versions 8 and 9 are read where Java reads 1 through 9. Happy to follow up on any of these separately if reviewers want them changed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
