JustinBinber opened a new pull request, #11176: URL: https://github.com/apache/paimon/pull/11176
### Purpose Spark Data Evolution `MERGE INTO` statements with both UPDATE and INSERT can scan and match touched target rows separately for each action. Existing source persistence and exchange reuse do not share the complete action matching result. Add `data-evolution.merge-into.join-reuse`, default `false`. Eligible statements share one projected, executor-local `DISK_ONLY` full outer match result. UPDATE retains its RowID-range partitioning and ordering; INSERT retains adaptive output sizing. The existing file-pruning pass and single atomic commit remain. The shared result is cleared on success and failure, including failures during commit. The option requires AQE and adaptive coalescing. Nondeterministic expressions, subqueries, DELETE, NOT MATCHED BY SOURCE and dedicated BLOB/VECTOR fields use the existing path. Full outer matching can replace cheaper broadcast-capable joins and adds shuffle and local I/O, so enable it after measuring a representative workload. It is not an incremental Driver-planning/OOM solution or a production disk quota. #### Complete SQL measurements Fixed clean baseline `b7cfb397de334d56b8c3eaeb60a2cfa9c9cab627`, candidate disabled and candidate enabled; three fresh processes/private physical copies per mode, balanced UFR/FRU/RUF order. Spark 3.5.9 on one 4-core/16-GiB host with two executor JVMs, warm OS cache. Generation, ALTER setup, independent validation, backups and parsing are outside the complete MERGE timing; shared materialization and cleanup are inside it. No completed sample was selectively rerun. | Median complete SQL | Upstream | Disabled | Enabled | | --- | ---: | ---: | ---: | | 65,536-file narrow fixture | 148.628 s | 150.415 s | 124.402 s | | 32,768-file, 259-column fixture; update 128 columns | 183.419 s | 183.834 s | 144.101 s | Enabled SQL time is 16.30% / 21.44% lower; disabled differs by +1.20% / +0.23%. Actual stage/task identities show one shared matching materialization per enabled execution. The pruning pass remains, so target scans decrease from three to two. All 18 independent validations pass: full row multisets, source immutability, old key/RowID pairs, complete and inserted RowID domains, seed and Snapshot checks. Actual Parquet footers show two INSERT files in every arm, with unchanged partial-update file counts and bytes. Resource tradeoffs, narrow / wide: - Task CPU decreases 16.06% / 24.37%; sampled common-live process CPU decreases 22.02% / 16.71%. - Shuffle written increases 71.90% / 95.29%; sampled process writes increase 23.12% / 16.64%. - Sampled concurrent three-JVM RSS decreases 11.44% narrow, increases 9.92% wide. - INSERT bytes increase 34.26% / 19.62%, despite equal file counts. Shared caches are 1,596,407 / 36,727,073 bytes in two blocks. These are descriptive single-host measurements, not multi-node network or statistical-significance claims. Process I/O includes logs/runtime/shuffle and is not physical-device or cache-only I/O. No Spark spill does not mean no disk writes. DISK_ONLY still needs transient materialization/operator memory, and cache size grows with retained match rows and width. No safety guarantee beyond the measured sizes or for skewed partitions is established. Unpersist requests precede SQL end; exact physical deletion at API return is not measured. Reproduction helper, frozen protocol, exact commands, all individual SQL samples and six resource/execution reports: [frozen reproduction bundle](https://gist.github.com/JustinBinber/3b2d75f5378edb4e340470b845ad7f02). The frozen-base patch reproduces the measured implementation; the PR integrates onto master `9b6c997` with unchanged production optimization bytes. A rejected prototype amplified INSERT files from 2 to 200 and is excluded from these measurements. ### Tests - Existing focused five-case Spark 3 suite, real 32-file independent oracle and both nine-arm cohorts. - Normal Spark 4.0/4.1 reactor build and test-source compilation. A preserved initial failure exposed stale Scala 2.12 output; a clean rebuild resolved it without production changes. - Exact Spark 4.0.5 API descriptors and actual native-cache runtime: alias ExprId preservation, materialization and persistent-RDD restoration after blocking cleanup. - Latest-master partition-pruning integration compiles with normal checks. Three focused Spark 4.0.5 runtime cases pass: concurrent update conflict without partial INSERT publication; stale-range rewrite after compaction with atomic INSERT; exact duplicate-match error and actual DISK_ONLY cache cleanup. - Independent code/test/document review found no blocking issue. The compaction case exercises precommit rewrite; no dedicated catch-loop retry execution is claimed. The existing committer retry algorithm is unchanged and operates on already-produced messages. - Generated configuration documentation and final checks: passed via `mvn -Pspark4,flink1,generate-docs -pl paimon-docs -nsu -DskipTests -am package` with normal checks. The new generated option row is included exactly; unrelated pre-existing ordering churn is excluded.. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
