JunRuiLee opened a new pull request, #565:
URL: https://github.com/apache/paimon-rust/pull/565
### Purpose
`DataSplit` already supports serialization to the Java-native wire format —
`DataSplit::serialize()` (raw v8, byte-compatible with Java
`DataSplit#serialize`) and `DataSplit::serialize_split_v1()` (the
`SplitSerializer` v1 frame), both golden-verified — but there is no inverse.
paimon-rust cannot reconstruct a `DataSplit` from bytes.
This blocks the planning-vs-reading separation where paimon-rust is the
**reader**: a planner in another process/engine serializes a `DataSplit` and
ships it to a worker, which must deserialize it before reading. This PR adds
that direction, so paimon-rust can serve as a read kernel behind an external
planner.
### Brief change log
- **core**: add the inverse of the existing serialization, built on existing
primitives (`BinaryRow::from_serialized_bytes` + typed getters) — no new
low-level infra:
- `DataFileMeta::from_serialized_row_data` — reverse of
`to_serialized_row_data` (fixed 20-field `BinaryRow`).
- `DataSplit::deserialize` — reverse of `serialize()` (raw v8 body);
consumes the whole buffer (trailing bytes rejected).
- `DataSplit::deserialize_split_v1` — reverse of `serialize_split_v1()`
(`SplitSerializer` frame): type 1 `DataSplit` and type 3 `IndexedSplit` (with
row ranges); other type ids return `Unsupported`.
- Leaf decoders: binary arrays (string/bigint), `SimpleStats`, Java
modified-UTF, deletion-file list, and big-endian cursor readers.
- **C binding**: `paimon_plan_from_split_bytes(data, len)` — build a
one-split plan from raw-v8 `DataSplit` bytes, wrapped in the existing
`paimon_plan` (usable by `paimon_table_read_to_arrow`, freed by
`paimon_plan_free`).
- Malformed/truncated input yields typed errors (`DataInvalid` /
`Unsupported`) and never panics; count/length prefixes are bounded against the
remaining buffer to avoid unbounded allocation. v8 only for now (current Java
`DataSplit.VERSION`), structured so a future v9 branch is a localized addition.
### Tests
- `cargo test -p paimon` — round-trip against the existing `datasplit_v8` /
`split_v1_data` / `split_v1_indexed` goldens, symmetric
`deserialize(serialize(x)) == x`, Java modified-UTF edge cases (NUL, non-ASCII,
surrogate pairs, malformed continuation bytes, truncation), unsupported type
ids, inverted row ranges, trailing bytes, and huge-count anti-abort cases.
- `cargo test -p paimon-c` — `paimon_plan_from_split_bytes` round-trip
(serialize a planned split, deserialize via the C entry, read), plus null/empty
and garbage error paths.
### API and Format
- Adds core methods and one C API symbol; no change to existing symbols or
to any storage format. Deserialization is byte-compatible with the existing
serialization.
### Documentation
- Doc comments on each new function; no separate docs needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
