JingsongLi opened a new pull request, #10414: URL: https://github.com/apache/paimon/pull/10414
### Purpose Use Native execution for PyPaimon's existing `upsert_by_key` inputs whose named `GenericRow` values have different field sets. Depends on https://github.com/apache/paimon-rust/pull/1061. Keep this PR as a draft until the Rust change is merged; Native CI continues to build Rust `main`. The adapter currently falls back to Python when rows have different supplied fields. Encode consecutive rows with the same field set as an Arrow RecordBatch and submit the ordered batches to Rust. This preserves missing fields, explicit NULL values and last-write-wins order without adding matching or update logic to the adapter. Rust core owns key matching, deduplication and column validation. Matched rows use the update projection; appended rows retain their actual write type, following Java's underlying append/Data Evolution write semantics. The public Python API is unchanged. ### Tests Added 20 cases exercising both Python and Native paths in batch and stream mode: - Different field sets and source field orders. - Last-write-wins across field sets, including superseded rows missing update columns. - Fan-out to duplicate target rows and ignored extra matched fields. - Explicit NULL versus absent fields, with append write-column metadata assertions. - Different append field sets in different partitions; partition keys implicitly distinguish matches. - Missing winning update columns and inconsistent surviving append shapes; errors leave snapshots and visible data unchanged. Native cases explicitly reject Python matching and per-partition upsert fallback. Local regression verification using the companion Rust binding: ```text table_upsert_by_key_test.py native_upsert_row_shapes_test.py table_update_test.py table_update_by_row_id_chunked_test.py native_upsert_capabilities_test.py native_nested_update_test.py native_blob_update_test.py native_update_test.py ``` With Native plan/read/write enabled: **991 passed, 2 skipped, 36 subtests passed**. Native execution counters: 1112 plans, 1345 reads, 760 writes. The complete Rust Python binding suite passed 321 tests. Flake8 for both changed Python files and diff checks passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
