JustinBinber opened a new pull request, #11176:
URL: https://github.com/apache/paimon/pull/11176

   ### Purpose
   
   Spark Data Evolution `MERGE INTO` statements with both UPDATE and INSERT can 
scan and match touched target rows separately for each action. Existing source 
persistence and exchange reuse do not share the complete action matching result.
   
   Add `data-evolution.merge-into.join-reuse`, default `false`. Eligible 
statements share one projected, executor-local `DISK_ONLY` full outer match 
result. UPDATE retains its RowID-range partitioning and ordering; INSERT 
retains adaptive output sizing. The existing file-pruning pass and single 
atomic commit remain. The shared result is cleared on success and failure, 
including failures during commit.
   
   The option requires AQE and adaptive coalescing. Nondeterministic 
expressions, subqueries, DELETE, NOT MATCHED BY SOURCE and dedicated 
BLOB/VECTOR fields use the existing path. Full outer matching can replace 
cheaper broadcast-capable joins and adds shuffle and local I/O, so enable it 
after measuring a representative workload. It is not an incremental 
Driver-planning/OOM solution or a production disk quota.
   
   #### Complete SQL measurements
   
   Fixed clean baseline `b7cfb397de334d56b8c3eaeb60a2cfa9c9cab627`, candidate 
disabled and candidate enabled; three fresh processes/private physical copies 
per mode, balanced UFR/FRU/RUF order. Spark 3.5.9 on one 4-core/16-GiB host 
with two executor JVMs, warm OS cache. Generation, ALTER setup, independent 
validation, backups and parsing are outside the complete MERGE timing; shared 
materialization and cleanup are inside it. No completed sample was selectively 
rerun.
   
   | Median complete SQL | Upstream | Disabled | Enabled |
   | --- | ---: | ---: | ---: |
   | 65,536-file narrow fixture | 148.628 s | 150.415 s | 124.402 s |
   | 32,768-file, 259-column fixture; update 128 columns | 183.419 s | 183.834 
s | 144.101 s |
   
   Enabled SQL time is 16.30% / 21.44% lower; disabled differs by +1.20% / 
+0.23%. Actual stage/task identities show one shared matching materialization 
per enabled execution. The pruning pass remains, so target scans decrease from 
three to two. All 18 independent validations pass: full row multisets, source 
immutability, old key/RowID pairs, complete and inserted RowID domains, seed 
and Snapshot checks. Actual Parquet footers show two INSERT files in every arm, 
with unchanged partial-update file counts and bytes.
   
   Resource tradeoffs, narrow / wide:
   
   - Task CPU decreases 16.06% / 24.37%; sampled common-live process CPU 
decreases 22.02% / 16.71%.
   - Shuffle written increases 71.90% / 95.29%; sampled process writes increase 
23.12% / 16.64%.
   - Sampled concurrent three-JVM RSS decreases 11.44% narrow, increases 9.92% 
wide.
   - INSERT bytes increase 34.26% / 19.62%, despite equal file counts. Shared 
caches are 1,596,407 / 36,727,073 bytes in two blocks.
   
   These are descriptive single-host measurements, not multi-node network or 
statistical-significance claims. Process I/O includes logs/runtime/shuffle and 
is not physical-device or cache-only I/O. No Spark spill does not mean no disk 
writes. DISK_ONLY still needs transient materialization/operator memory, and 
cache size grows with retained match rows and width. No safety guarantee beyond 
the measured sizes or for skewed partitions is established. Unpersist requests 
precede SQL end; exact physical deletion at API return is not measured.
   
   Reproduction helper, frozen protocol, exact commands, all individual SQL 
samples and six resource/execution reports: [frozen reproduction 
bundle](https://gist.github.com/JustinBinber/3b2d75f5378edb4e340470b845ad7f02). 
The frozen-base patch reproduces the measured implementation; the PR integrates 
onto master `9b6c997` with unchanged production optimization bytes. A rejected 
prototype amplified INSERT files from 2 to 200 and is excluded from these 
measurements.
   
   ### Tests
   
   - Existing focused five-case Spark 3 suite, real 32-file independent oracle 
and both nine-arm cohorts.
   - Normal Spark 4.0/4.1 reactor build and test-source compilation. A 
preserved initial failure exposed stale Scala 2.12 output; a clean rebuild 
resolved it without production changes.
   - Exact Spark 4.0.5 API descriptors and actual native-cache runtime: alias 
ExprId preservation, materialization and persistent-RDD restoration after 
blocking cleanup.
   - Latest-master partition-pruning integration compiles with normal checks. 
Three focused Spark 4.0.5 runtime cases pass: concurrent update conflict 
without partial INSERT publication; stale-range rewrite after compaction with 
atomic INSERT; exact duplicate-match error and actual DISK_ONLY cache cleanup.
   - Independent code/test/document review found no blocking issue. The 
compaction case exercises precommit rewrite; no dedicated catch-loop retry 
execution is claimed. The existing committer retry algorithm is unchanged and 
operates on already-produced messages.
   - Generated configuration documentation and final checks: passed via `mvn 
-Pspark4,flink1,generate-docs -pl paimon-docs -nsu -DskipTests -am package` 
with normal checks. The new generated option row is included exactly; unrelated 
pre-existing ordering churn is excluded..
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to