JingsongLi opened a new pull request, #10418: URL: https://github.com/apache/paimon/pull/10418
### Purpose Run PyPaimon's existing single-process data-evolution `merge_into` operation through Rust core for eligible native tables. Depends on https://github.com/apache/paimon-rust/pull/1068. This PR is draft until that runtime capability reaches Rust main. The native CI job continues installing Rust main. - Forward existing Arrow/pandas and self-merge inputs, normalized matched UPDATE/DELETE and unmatched INSERT clauses to native batch/stream TableUpdate. - Let Rust own snapshot pinning, key matching, SQL conditions (including subqueries), ordered action selection and prepared-file staging. Python retains its public WhenMatched/WhenNotMatched API and commit-message bridge. - Align Python fallback with Paimon Spark: validate source-match cardinality before action predicates and allow duplicate source matches for a sole unconditional DELETE. - Select Python execution before starting native work for non-self table sources, unsupported assignments/key layouts and packed-video inserts. Native errors propagate without Python retry or deleting earlier prepared files. - Add a native MERGE execution counter to the existing CI gates and end-to-end tests that reject Python MERGE orchestration on the native path. ### Tests - Native planning/read/write/update/commit regression suite: 994 passed, 2 skipped, 42 subtests passed, using a local build of the paired Rust PR. - Native counters: plans=1149, reads=1359, writes=822, commits=147; updates row_id=189, grouped=54, predicate=91, upsert=120, incremental=55, merge=28. - MERGE tests cover batch/stream update-delete-insert, ordered clauses, NULL keys/conditions, duplicate source and target keys, renamed keys, pandas input, self-merge, empty targets, deferred literal errors, scalar/IN/correlated EXISTS conditions and unreferenced unreadable Blob payloads. - Paired Rust validation: core library 3,891 passed (6 ignored), DataFusion condition adapter 3 passed, Python binding suite 333 passed, Clippy 1.98.0 all-target/workspace with fulltext/vortex passed. - Changed Python files pass flake8 and Python 3.6 syntax parsing; `git diff --check` passes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
