JunRuiLee opened a new pull request, #745:
URL: https://github.com/apache/paimon-rust/pull/745
## What
`DataSplit::deserialize` accepted wire version 8 only, so any `DataSplit`
produced by
current Paimon Java is rejected with `Unsupported`.
Java bumped `DataSplit#VERSION` 8 → 9 in apache/paimon#9313 when it appended
`_WRITE_COLS_SEQUENCES` (per-column maximum sequence numbers) to
`DataFileMeta.SCHEMA`.
The split body itself is unchanged — the only difference is that the per-file
`DataFileMeta` row grew by one nullable trailing field:
| version | per-file row | Java serializer |
|---|---|---|
| 8 | 20 fields, ends at `_WRITE_COLS` |
`DataFileMetaWriteColsLegacySerializer` |
| 9 | 21 fields, `_WRITE_COLS_SEQUENCES` at slot 20 |
`DataFileMetaSerializer` |
## Changes
- `DataFileMeta` carries `column_max_sequence_numbers: Option<Vec<i64>>`.
- The row codec is layout-aware via `DataFileMetaRowLayout`. The two layouts
share
slots 0..=19 and differ only in arity, which the enclosing split's version
selects.
Only the current layout is written, mirroring Java keeping its legacy
serializers
deserialize-only.
- `DataSplit::deserialize` accepts v8 and v9; `serialize` emits v9.
- The field is decoded from manifest entries and declared in the manifest
Avro schema,
so a file read from a manifest and handed on inside a split no longer
silently drops it.
## Goldens
All taken from the Java compatibility fixtures rather than generated here:
- `goldens/datasplit_v9.bin` is the raw body of `compatibility/datasplit-v9`
(the Java
fixture is an object stream: `TC_BLOCKDATALONG` + declared length + body +
`TC_ENDBLOCKDATA`). Extracting the same way from
`compatibility/datasplit-v8` reproduces
the existing `goldens/datasplit_v8.bin` byte for byte, which is what
validates the
extraction.
- `goldens/split_v1_data.bin` / `split_v1_indexed.bin` are refreshed from
their Java
counterparts, whose embedded `DataSplit` is now v9. They were still
carrying a v8 body.
The v8 golden is kept as a legacy-decode test.
## Note for reviewers
`serialize` now emits v9 rather than v8. That follows Java, which no longer
writes v8, and
v8 stays readable — but it does mean bytes produced here can no longer be
read by a Paimon
older than apache/paimon#9313. Happy to gate it behind an explicit version
argument instead
if you would rather not move the write side in the same change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
