JustinBinber opened a new pull request, #11174:
URL: https://github.com/apache/paimon/pull/11174

   ### Purpose
   
   Separate key-based DELETE and INSERT statements on a Data Evolution table 
publish two snapshots. Readers can observe the deletion-only result, and an 
append failure leaves that result committed. Add `CALL 
sys.replace_by_keys(table, source_query, key_columns, target_filter)` to 
prepare deletion vectors and appended files using the existing paths and 
publish them together in one data commit.
   
   The procedure performs a key-only semi join for deletion and preserves every 
source row on append, including duplicates. It supports composite keys and 
ordinary SQL NULL equality, aligns all source columns by name using INSERT's 
assignment rules, and compares original source keys before INSERT casts. 
`target_filter` restricts deletion only. It does not execute native MERGE or 
reconstruct old target payload columns.
   
   Paimon source data snapshots are bound across the two preparation jobs while 
preserving the analyzed schema and effective relation options. Commands, 
streaming/nondeterministic inputs, incremental/monitoring scans, and plans that 
cannot safely repeat under different projections are rejected before writing; 
those batches require explicit materialization. Known returned messages are 
aborted on preparation failure, but publication exceptions are not blindly 
aborted or retried. This is single-table atomic visibility with existing 
conflict checks, not general SQL transactions or serializable business-key 
uniqueness. Inline compaction is disabled for this operation; the two sets of 
commit metadata can coexist in memory.
   
   ### Tests
   
   Added `ReplaceByKeysProcedureTest` coverage for composite keys, 
NULLs/duplicates, name alignment, one snapshot, old/new RowIDs, actual append 
execution failure, rejecting source DML without side effects, original-key 
comparison before INSERT casts, Spark-compatible Unicode column matching, added 
columns after the last source write, and effective relation scan options.
   
   Local validation used Java 8, Spark 3.5.9 / Scala 2.12, and clean community 
base `b7cfb397de334d56b8c3eaeb60a2cfa9c9cab627`:
   
   - Changed production sources compiled to Java 8 bytecode; public CALL 
functional probes passed. The final source-binding checks and all eight final 
performance samples used the same frozen candidate overlay.
   - Scoped Spotless check passed without `fast-build`; `git diff --check` 
passed.
   - Full local Maven validation/compilation was attempted but blocked by 
missing local `2.2-SNAPSHOT` test artifacts before reaching the changed Spark 
module. The standalone probes are not reported as a Maven/ScalaTest suite pass. 
This PR remains a draft pending normal CI verification.
   
   Local A/B compares **both real DELETE and INSERT together** against the 
**whole public CALL**, including analysis, preparation and commit. Both arms 
use the same physical 121-column source/target data, fresh private input copies 
and JVMs, Java 8, a 768 MiB heap, `local[2]`, two fixed logical CPUs and two 
shuffle partitions. Order is A1, B1, B2, A2 at each size. Fixture generation, 
copying and full correctness checks are outside the SQL timer.
   
   | Real base files | DELETE+INSERT A1 / A2 | Atomic CALL B1 / B2 | Mean B/A |
   |---:|---:|---:|---:|
   | 32 | 10.533 / 9.744 s | 10.538 / 10.202 s | 1.0229 |
   | 1024 | 16.831 / 16.186 s | 16.915 / 17.369 s | 1.0384 |
   
   Every sample passed the complete 121-column multiset, duplicate/NULL, 
untouched/new RowID and historical-snapshot checks. A published two snapshots 
and B one. No memory/disk spill occurred. Mean process CPU A/B was 16.60/18.35 
s at 32 files and 29.53/31.15 s at 1024; mean sampled peak heap was 259.0/259.9 
MiB and 265.9/272.7 MiB, respectively. Both paths still shuffle keys.
   
   These are two samples per path/size on one local JVM per sample, with many 
tiny files; they establish neither statistical significance nor a production 
overhead bound or speedup. Host-memory admission paused after 32/A1 before B1 
had started; remaining unstarted slots resumed unchanged, and all samples and 
wait records were retained. No performance sample was retried. No cloud cluster 
or production throughput claim is made.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to