JingsongLi opened a new pull request, #10624: URL: https://github.com/apache/paimon/pull/10624
### Purpose Use Native execution for Arrow upsert inputs that PyPaimon already supports: large-string match keys, partition fields and nested values, plus equivalent fixed-size binary keys. The current exact type comparison sends these operations back to Python even though they represent the same logical table values. Depends on apache/paimon-rust#1144. Native CI continues building apache/paimon-rust `main`, so the Rust change must merge first. - Reuse the existing Arrow schema compatibility policy for supplied upsert columns. Core owns normalization, partition-scoped matching, last-write-wins deduplication and file writing. Numeric conversions, different timestamp zones and other non-equivalent types keep their existing dispatch behavior. - Reuse `read_arrow_schema` when combining DE column bunches. File readers already return runtime-nullable nested types; reapplying the stricter logical schema could make an otherwise valid read fail. - Reuse the core-backed write normalizer at the Python row-ID writer boundary to restore declared nested write types and reject visible NULLs in required children. Keep NULL parent ROW values valid. - Close both the mapped batch iterator and its underlying merged reader on validation or writing errors, even while the exception traceback is retained. - Add Native/Python parity tests covering batch/stream upserts, multiple Arrow chunks, nested ROW/ARRAY/MAP strings, duplicate and NULL keys, partition isolation, partial writes, row identity, REST-backed writes, and required nested fields in row-ID and predicate updates. The scope is read/write/update execution. There are no catalog metadata transport changes or new public APIs, options or dependencies. ### Tests Built the Rust bindings from apache/paimon-rust#1144 and ran with Native planning, reading, writing and committing enabled: - New layout cases plus chunked row-ID writer tests: 42 passed before the final reader-close regression was added. - Final regression selection (`native_upsert_layout_test`, `table_update_by_row_id_chunked_test`, `table_update_test`, `data_evolution_test`, `native_blob_update_test`): 383 passed, 24 subtests passed. - Broader upsert/nested update/BLOB/DE/concatenated reader selection: 568 passed, 2 skipped, 24 subtests passed. - Flake8 for every changed Python file and `git diff --check`. - The original Native fallback and core LargeUtf8 rejection were reproduced before the fixes. The new reader-close test failed in both validation-error and writer-error cases before explicitly closing the inner iterator. - Independent review of type eligibility, nested constraints and resource ownership; the identified reader-close issue was repaired and re-reviewed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
