JingsongLi opened a new pull request, #536: URL: https://github.com/apache/paimon-rust/pull/536
## What changed - add Java-compatible MAP shared-shredding write and read plans for `MAP<STRING, T>` - infer the number of shared physical columns from the first buffered row - persist and restore the field dictionary, column allocation, overflow set, and layout metadata in Parquet - integrate MAP and Variant shredding through shared read/write plan abstractions - assemble logical MAP values before residual predicate evaluation - support `none`, `lz4`, and `zstd` field-dictionary compression - normalize nested Arrow value types and gather only mapped values with `interleave` - reject malformed or oversized shredding metadata before allocation or schema construction ## Why Paimon Java supports the PIP-43 MAP shared-shredding layout, but the Rust implementation previously supported only Variant shredding. Rust therefore could not write or reconstruct the Java-compatible MAP physical layout. The read path also needs to account for Java-generated nested Arrow field metadata and untrusted footer dimensions without materializing every shared physical value column. ## Impact Parquet readers and writers can use: ``` fields.<column>.map.storage-layout=shared-shredding fields.<column>.map.shared-shredding.max-columns=<n> ``` The implementation preserves null and empty MAP semantics, overflow entries, nested ARRAY/MAP values, and residual-filter ordering. Unsupported simultaneous Variant and MAP shredding plans are rejected. ## Validation - `cargo fmt --all -- --check` - `cargo clippy -p paimon --lib --tests --no-deps -- -D warnings` - `cargo test -p paimon --lib` — 1625 passed, 1 ignored -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
