JunRuiLee opened a new pull request, #9386:
URL: https://github.com/apache/paimon/pull/9386
### Purpose
`BucketVectorSearchSplit` currently has no public serialized form: it is only
Java-serializable via `writeObject`, which is not a stable contract and is
not
reachable from a non-JVM reader.
This adds `BucketVectorSearchSplitSerializer`, a versioned big-endian binary
format for it, so an engine can hand a bucket-scoped vector-search split to a
reader in another language. The layout frames the split's `DataSplit` with
its
own `serialize` bytes and then writes the per-file vector payload metadata
and
the per-file inclusive row ranges:
```
i64 magic = "PKVSPLIT"; i32 version = 1
bytes dataSplit = i32 length + DataSplit.serialize bytes
i32 payloadCount
payload = string indexType, string fileName, i64 fileSize, i64 rowCount,
optional string externalPath, i64 rowRangeStart, i64 rowRangeEnd,
i32 indexFieldId, optional i32[] extraFieldIds,
optional bytes indexMeta, bytes sourceMeta
i32 rangeFileCount; (string fileName, i32 rangeCount, (i64 from, i64 to)[])*
```
Two deliberate choices, since the consumer is not necessarily a JVM:
- strings and byte arrays use an `i32` length plus standard UTF-8, not
`writeUTF`'s modified UTF-8 with its 64KB ceiling;
- optionals use an explicit 1-byte presence flag, and a value other than 0
or 1
is rejected rather than coerced.
Inputs that the format cannot represent are rejected at serialization time
rather than written out lossily: negative file size or row count, a non-empty
`dvRanges()`, a missing `globalIndexMeta` or `sourceMeta`, duplicate payload
file names, and a row range whose start exceeds its end.
Read-side only; nothing in the existing write or planning path changes.
### Tests
`BucketVectorSearchSplitSerializerTest` covers the round trip, each rejected
input, and a byte-level check against a stored fixture
(`compatibility/bucket-vector-search-split-v1.base64`) so the layout cannot
change silently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
