JingsongLi opened a new pull request, #1236: URL: https://github.com/apache/paimon-rust/pull/1236
### Purpose Upgrade the Rust integration from DataFusion 54 to 55.2.0 and keep Python DataFusion 55 interoperability working. ### Brief change log - Upgrade DataFusion/FFI to 55.2.0, Arrow/Parquet to 59, ORC to 0.9, and PyO3 to 0.29; use Python DataFusion 55.0.0 and refresh dependency reports. - Migrate execution-plan statistics, child replacement, expression traversal, and planner APIs to DataFusion 55. - Bridge Mosaic and Vortex's Arrow 58 APIs through the Arrow C Data Interface, retaining buffer ownership without copying payloads. - Keep scalar floating-zero comparisons with DataFusion so native Paimon pushdown cannot drop the other sign of zero, including completed dynamic filters. - Preserve nullable multiset elements with a marked Arrow list of element/count structs. Retain legacy Map write inputs, schema evolution, Java Row/Avro encodings, and Parquet MAP storage; reject malformed entries with an error instead of panicking. ### Tests On the final implementation: - Rust workspace library tests: 5,346 passed, 8 ignored. - Related Rust/DataFusion integration tests: 207 passed, including unfiltered native DataFusion comparisons for FLOAT/DOUBLE signed zero, partition pruning, and Mosaic/Vortex reads and writes. - Python tests after rebuilding the extension: 650 passed. - Regression coverage for completed dynamic filters, Arrow bridge ownership, raw Java Row bytes, nullable multiset Avro round trips, legacy Parquet schemas, hidden/null/sliced children, schema evolution, and malformed planner input. - Rust 1.98.0 workspace Clippy with all targets and the Vortex feature, denying warnings; formatting and dependency-report verification passed. Local validation ran on macOS; Linux/Windows and MSRV builds are left to CI. ### API and Format DataFusion consumers must use 55.x and Arrow consumers must use 59.x. `MULTISET<T>` now uses `List<Struct<key: T, value: Int32>>` with `paimon.multiset=true` entry-field metadata, preserving NULL elements without violating Arrow 59's non-null Map-key requirement. Existing non-null-key Arrow Map inputs remain accepted. Existing Row, Avro, and Parquet disk encodings remain readable; new Parquet writes keep MAP storage and reject visible NULL multiset elements. Mosaic and Vortex continue to reject multiset columns. ### Documentation Update the SQL dependency examples and document the multiset Arrow representation and format constraints. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
