JingsongLi opened a new pull request, #973: URL: https://github.com/apache/paimon-rust/pull/973
### Purpose Complete the core support needed to run PyPaimon's nested ARRAY/MAP/ROW updates and upserts through Native execution. An upsert currently rejects equivalent Arrow schemas such as `list<item: int>` versus Paimon's `list<element: int>`. Removing the Python nested-column guard also exposes missing row-ID constructor conversions and differences in safe ROW casts. ### Brief change log - Normalize complete upsert batches with the ordinary write normalizer before key matching or staging files. - Normalize MAP children by key/value position, including custom Arrow child names. - Implement row-ID constructor conversions for top-level lists of pairs to MAP, ROW to MAP, and MAP to ROW in Rust core. Preserve slices, empty collections, NULL parents and nested NULL values. - Retain the distinction between safe predicate assignments and whole-column row-ID constructor fallback. Validate ROW field order and nullability, reject ambiguous top-level dict NULL values, and support dictionary logical NULLs and list/view layouts. - Add public core upsert tests for duplicate-key matching, appends, untouched fields and failure cleanup, plus conversion regressions. All behavior remains in the Rust core; the Python binding API is unchanged. Invalid UTF-8 MAP field names are rejected rather than reproducing PyArrow's silent NULL result for malformed names. ### Tests - `cargo test --locked -p paimon --lib table::update_input::tests` — 28 passed. - `cargo test --locked -p paimon --lib table::write_batch_normalize::tests` — 4 passed. - `cargo test --locked -p paimon --test table_update_test --test table_update_nested_test --test table_update_paths_test` — 30 passed. - `cargo clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings` and `cargo fmt --all --check` passed. - Built the Python extension and ran the dependent PyPaimon update/upsert/write regression suite with PyArrow 18.1 and all five Native CI flags: 779 passed, 2 skipped, 16 subtests passed. The two skips are Python-only MAP view-key cases without an Arrow `take` kernel; their Native counterparts pass. ### API and Format No public API or persisted format changes. Upserts accept the same equivalent nested layouts as ordinary writes. ### Documentation No new configuration or public methods. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
