JustinBinber opened a new pull request, #11174: URL: https://github.com/apache/paimon/pull/11174
### Purpose Separate key-based DELETE and INSERT statements on a Data Evolution table publish two snapshots. Readers can observe the deletion-only result, and an append failure leaves that result committed. Add `CALL sys.replace_by_keys(table, source_query, key_columns, target_filter)` to prepare deletion vectors and appended files using the existing paths and publish them together in one data commit. The procedure performs a key-only semi join for deletion and preserves every source row on append, including duplicates. It supports composite keys and ordinary SQL NULL equality, aligns all source columns by name using INSERT's assignment rules, and compares original source keys before INSERT casts. `target_filter` restricts deletion only. It does not execute native MERGE or reconstruct old target payload columns. Paimon source data snapshots are bound across the two preparation jobs while preserving the analyzed schema and effective relation options. Commands, streaming/nondeterministic inputs, incremental/monitoring scans, and plans that cannot safely repeat under different projections are rejected before writing; those batches require explicit materialization. Known returned messages are aborted on preparation failure, but publication exceptions are not blindly aborted or retried. This is single-table atomic visibility with existing conflict checks, not general SQL transactions or serializable business-key uniqueness. Inline compaction is disabled for this operation; the two sets of commit metadata can coexist in memory. ### Tests Added `ReplaceByKeysProcedureTest` coverage for composite keys, NULLs/duplicates, name alignment, one snapshot, old/new RowIDs, actual append execution failure, rejecting source DML without side effects, original-key comparison before INSERT casts, Spark-compatible Unicode column matching, added columns after the last source write, and effective relation scan options. Local validation used Java 8, Spark 3.5.9 / Scala 2.12, and clean community base `b7cfb397de334d56b8c3eaeb60a2cfa9c9cab627`: - Changed production sources compiled to Java 8 bytecode; public CALL functional probes passed. The final source-binding checks and all eight final performance samples used the same frozen candidate overlay. - Scoped Spotless check passed without `fast-build`; `git diff --check` passed. - Full local Maven validation/compilation was attempted but blocked by missing local `2.2-SNAPSHOT` test artifacts before reaching the changed Spark module. The standalone probes are not reported as a Maven/ScalaTest suite pass. This PR remains a draft pending normal CI verification. Local A/B compares **both real DELETE and INSERT together** against the **whole public CALL**, including analysis, preparation and commit. Both arms use the same physical 121-column source/target data, fresh private input copies and JVMs, Java 8, a 768 MiB heap, `local[2]`, two fixed logical CPUs and two shuffle partitions. Order is A1, B1, B2, A2 at each size. Fixture generation, copying and full correctness checks are outside the SQL timer. | Real base files | DELETE+INSERT A1 / A2 | Atomic CALL B1 / B2 | Mean B/A | |---:|---:|---:|---:| | 32 | 10.533 / 9.744 s | 10.538 / 10.202 s | 1.0229 | | 1024 | 16.831 / 16.186 s | 16.915 / 17.369 s | 1.0384 | Every sample passed the complete 121-column multiset, duplicate/NULL, untouched/new RowID and historical-snapshot checks. A published two snapshots and B one. No memory/disk spill occurred. Mean process CPU A/B was 16.60/18.35 s at 32 files and 29.53/31.15 s at 1024; mean sampled peak heap was 259.0/259.9 MiB and 265.9/272.7 MiB, respectively. Both paths still shuffle keys. These are two samples per path/size on one local JVM per sample, with many tiny files; they establish neither statistical significance nor a production overhead bound or speedup. Host-memory admission paused after 32/A1 before B1 had started; remaining unstarted slots resumed unchanged, and all samples and wait records were retained. No performance sample was retried. No cloud cluster or production throughput claim is made. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
