JingsongLi opened a new pull request, #1411:
URL: https://github.com/apache/paimon-rust/pull/1411
### Purpose
Remove Rust-side compatibility blockers behind PyPaimon's fallback for
fixed-bucket append, shredded VARIANT writes, ORC shared-shredding MAP reads,
and Data Evolution partial-column commits.
### Brief change log
- Allocate append sequence bounds per partition/bucket across file rolling
and repeated prepares; restore from the selected snapshot and reset for
overwrite.
- Preserve incoming Arrow field IDs and write Java-compatible nested VARIANT
wrapper and LIST-element IDs into both the Parquet schema and Arrow footer.
Keep runtime Arrow schemas and ordinary ROW siblings unchanged.
- Recover ORC MAP shredding annotations from optional Arrow IPC metadata and
assemble logical MAP values before residual filtering. Extract annotations
without interpreting unrelated Arrow types, avoiding metadata-induced panics.
- Accept dedicated column-file ranges contained within one normal anchor in
the same partition/bucket. Reject nonpositive and overflowing ranges; ordinary
column files still require an exact normal anchor.
### Tests
- Core unit suite: **4,883 passed, 7 ignored, 1 filtered**:
`cargo test --locked -p paimon --lib --no-default-features --features
storage-memory,storage-fs -- --quiet --test-threads=4 --skip
io::cache::disk::shared::tests::process_exit_releases_staging_for_cleanup`
The excluded existing subprocess/cache test fails intermittently in the
full suite and passes when run separately.
- **37 integration tests passed** across `blob_update_test`,
`data_evolution_variant_sidecar_test`, `row_variant_test`,
`nested_schema_evolution_test`, and `nested_native_write_compat_test`, with the
same storage features.
- `cargo fmt --all -- --check` and `git diff --check` passed.
- `cargo clippy --locked -p paimon --all-targets --features vortex -- -D
warnings -A clippy::nonminimal_bool` passed. Strict Clippy is blocked by six
pre-existing `nonminimal_bool` diagnostics outside this change.
- Reproduced the missing-ID writer failure, malformed IPC panics, and
invalid row-count commit before their fixes. Added regressions for those cases
and ordinary ROW type preservation.
- Compared Java/PyPaimon source at `8ad61142f3`; the unmodified PyPaimon MAP
assembler also read the ORC fixture successfully. Java engine integration tests
were not run.
### API and Format
No new public API or storage format. Correct existing append metadata and
Java-compatible Parquet field-ID annotations, and read the existing ORC
shared-shredding layout.
### Documentation
PyPaimon's four native-qualification guards remain unchanged in its separate
repository and require a follow-up that retains fallback for older runtimes.
Implementation and review were AI-assisted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]