JingsongLi opened a new pull request, #1411:
URL: https://github.com/apache/paimon-rust/pull/1411

   ### Purpose
   
   Remove Rust-side compatibility blockers behind PyPaimon's fallback for 
fixed-bucket append, shredded VARIANT writes, ORC shared-shredding MAP reads, 
and Data Evolution partial-column commits.
   
   ### Brief change log
   
   - Allocate append sequence bounds per partition/bucket across file rolling 
and repeated prepares; restore from the selected snapshot and reset for 
overwrite.
   - Preserve incoming Arrow field IDs and write Java-compatible nested VARIANT 
wrapper and LIST-element IDs into both the Parquet schema and Arrow footer. 
Keep runtime Arrow schemas and ordinary ROW siblings unchanged.
   - Recover ORC MAP shredding annotations from optional Arrow IPC metadata and 
assemble logical MAP values before residual filtering. Extract annotations 
without interpreting unrelated Arrow types, avoiding metadata-induced panics.
   - Accept dedicated column-file ranges contained within one normal anchor in 
the same partition/bucket. Reject nonpositive and overflowing ranges; ordinary 
column files still require an exact normal anchor.
   
   ### Tests
   
   - Core unit suite: **4,883 passed, 7 ignored, 1 filtered**:
     `cargo test --locked -p paimon --lib --no-default-features --features 
storage-memory,storage-fs -- --quiet --test-threads=4 --skip 
io::cache::disk::shared::tests::process_exit_releases_staging_for_cleanup`
     The excluded existing subprocess/cache test fails intermittently in the 
full suite and passes when run separately.
   - **37 integration tests passed** across `blob_update_test`, 
`data_evolution_variant_sidecar_test`, `row_variant_test`, 
`nested_schema_evolution_test`, and `nested_native_write_compat_test`, with the 
same storage features.
   - `cargo fmt --all -- --check` and `git diff --check` passed.
   - `cargo clippy --locked -p paimon --all-targets --features vortex -- -D 
warnings -A clippy::nonminimal_bool` passed. Strict Clippy is blocked by six 
pre-existing `nonminimal_bool` diagnostics outside this change.
   - Reproduced the missing-ID writer failure, malformed IPC panics, and 
invalid row-count commit before their fixes. Added regressions for those cases 
and ordinary ROW type preservation.
   - Compared Java/PyPaimon source at `8ad61142f3`; the unmodified PyPaimon MAP 
assembler also read the ORC fixture successfully. Java engine integration tests 
were not run.
   
   ### API and Format
   
   No new public API or storage format. Correct existing append metadata and 
Java-compatible Parquet field-ID annotations, and read the existing ORC 
shared-shredding layout.
   
   ### Documentation
   
   PyPaimon's four native-qualification guards remain unchanged in its separate 
repository and require a follow-up that retains fallback for older runtimes.
   
   Implementation and review were AI-assisted.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to