JustinBinber opened a new issue, #11323: URL: https://github.com/apache/paimon/issues/11323
### Search before asking - [x] I searched existing Paimon issues for narrow sources, MERGE wide payloads, shared joins, join reuse and cache costs, and did not find an issue for this specific optimization. ### Motivation Spark MERGE INTO on a Data Evolution append target can share one Full Outer Join between its UPDATE and INSERT branches. The shared result currently includes source fields that are needed only for INSERT. If the source has a large INSERT-only payload, matched rows carry that payload through shuffle and the executor disk cache even though UPDATE does not use it. The target here is Data Evolution append mode: no primary key, bucket=-1 and row tracking enabled. This proposal is about shared-join payload cost, not primary-key LSM merging or a general solution to Driver OOM. ### Solution Add an opt-in, default-off narrow-source path alongside the existing shared-join option: 1. Share the matching and UPDATE dependencies, together with row-presence markers. 2. Exclude INSERT-only source fields from that shared result. 3. Get unmatched source keys from the shared result, then use a null-safe LeftSemi membership join against a reread of the **same pinned source snapshot** to obtain INSERT payloads. Preserve duplicate unmatched source rows and ordinary MERGE NULL semantics. 4. Initially restrict this to eligible Paimon append source plans with a single same-type INT/BIGINT equality key. Retain the existing path for unsupported plans or when there are no source fields to exclude. Users should not have to promise that a source is stable. This still scans the source for payload reread. DE-source positional RowID reread is a separate experiment and is outside this issue and PR. ### Evidence and tradeoffs A bounded synthetic screen used one 4-core/16-GiB host with two Executor JVMs, Spark 3.5.9, AQE enabled, a 32,768-file/259-column target, and a 39,321-row source. UPDATE assigns 128 columns. The wide source has a genuinely stored logical 8-KiB INSERT-only field. For approximately 90% INSERT, existing/narrow sharing observed: | Metric | Existing sharing | Narrow sharing | |---|---:|---:| | Complete MERGE SQL | 76.95 s | 72.63 s | | Shuffle written | 316.79 MB | 10.77 MB | | Peak shared disk cache | 312.20 MB | 18.78 MB | | Sampled process-tree RSS | 3.78 GiB | 4.80 GiB | Each arm/scenario has only one sample: this is **not** a stable speedup or statistical significance claim. The memory observation is worse by about 27%. The no-INSERT-only-wide-field control used the fallback path and its narrow-enabled SQL observation was 2.21% slower; all-INSERT also fell back, with no shared cache/shuffle. These results support discussing a bounded, default-off option, not default enablement or a universal memory improvement. All nine complete SQL observations have independent full-column multiset, source immutability, RowID, snapshot and output-layout validation. [Synthetic generator/helper sources, exact SQL generators, all nine sample metrics, validation summaries, resource controls and limitations](https://github.com/JustinBinber/paimon/tree/0c5bd5e93b74dad1160813daa4f45d86c2db8756/benchmarks/merge-narrow-source) are public. This is one physical host, not a multi-node benchmark. FileIO returned bytes are not physical disk/network traffic. ### Anything else? Implementation proposal: #11322. The implementation and evidence are available for design review; community agreement or merge approval is not assumed. Related semantic tests cover snapshot pinning, safe fallback, NULLs, duplicate unmatched rows and cleanup on failure. Existing exact-source runs passed 17 tests each on Spark 3.5.9/Scala 2.12/JDK 8 and Spark 4.1.4/Scala 2.13/JDK 17; current-master integration CI is tracked on the PR. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR. The proposed implementation is #11322. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
