JustinBinber opened a new issue, #11323:
URL: https://github.com/apache/paimon/issues/11323

   ### Search before asking
   
   - [x] I searched existing Paimon issues for narrow sources, MERGE wide 
payloads, shared joins, join reuse and cache costs, and did not find an issue 
for this specific optimization.
   
   ### Motivation
   
   Spark MERGE INTO on a Data Evolution append target can share one Full Outer 
Join between its UPDATE and INSERT branches. The shared result currently 
includes source fields that are needed only for INSERT. If the source has a 
large INSERT-only payload, matched rows carry that payload through shuffle and 
the executor disk cache even though UPDATE does not use it.
   
   The target here is Data Evolution append mode: no primary key, bucket=-1 and 
row tracking enabled. This proposal is about shared-join payload cost, not 
primary-key LSM merging or a general solution to Driver OOM.
   
   ### Solution
   
   Add an opt-in, default-off narrow-source path alongside the existing 
shared-join option:
   
   1. Share the matching and UPDATE dependencies, together with row-presence 
markers.
   2. Exclude INSERT-only source fields from that shared result.
   3. Get unmatched source keys from the shared result, then use a null-safe 
LeftSemi membership join against a reread of the **same pinned source 
snapshot** to obtain INSERT payloads. Preserve duplicate unmatched source rows 
and ordinary MERGE NULL semantics.
   4. Initially restrict this to eligible Paimon append source plans with a 
single same-type INT/BIGINT equality key. Retain the existing path for 
unsupported plans or when there are no source fields to exclude. Users should 
not have to promise that a source is stable.
   
   This still scans the source for payload reread. DE-source positional RowID 
reread is a separate experiment and is outside this issue and PR.
   
   ### Evidence and tradeoffs
   
   A bounded synthetic screen used one 4-core/16-GiB host with two Executor 
JVMs, Spark 3.5.9, AQE enabled, a 32,768-file/259-column target, and a 
39,321-row source. UPDATE assigns 128 columns. The wide source has a genuinely 
stored logical 8-KiB INSERT-only field.
   
   For approximately 90% INSERT, existing/narrow sharing observed:
   
   | Metric | Existing sharing | Narrow sharing |
   |---|---:|---:|
   | Complete MERGE SQL | 76.95 s | 72.63 s |
   | Shuffle written | 316.79 MB | 10.77 MB |
   | Peak shared disk cache | 312.20 MB | 18.78 MB |
   | Sampled process-tree RSS | 3.78 GiB | 4.80 GiB |
   
   Each arm/scenario has only one sample: this is **not** a stable speedup or 
statistical significance claim. The memory observation is worse by about 27%. 
The no-INSERT-only-wide-field control used the fallback path and its 
narrow-enabled SQL observation was 2.21% slower; all-INSERT also fell back, 
with no shared cache/shuffle. These results support discussing a bounded, 
default-off option, not default enablement or a universal memory improvement.
   
   All nine complete SQL observations have independent full-column multiset, 
source immutability, RowID, snapshot and output-layout validation. [Synthetic 
generator/helper sources, exact SQL generators, all nine sample metrics, 
validation summaries, resource controls and 
limitations](https://github.com/JustinBinber/paimon/tree/0c5bd5e93b74dad1160813daa4f45d86c2db8756/benchmarks/merge-narrow-source)
 are public. This is one physical host, not a multi-node benchmark. FileIO 
returned bytes are not physical disk/network traffic.
   
   ### Anything else?
   
   Implementation proposal: #11322. The implementation and evidence are 
available for design review; community agreement or merge approval is not 
assumed. Related semantic tests cover snapshot pinning, safe fallback, NULLs, 
duplicate unmatched rows and cleanup on failure. Existing exact-source runs 
passed 17 tests each on Spark 3.5.9/Scala 2.12/JDK 8 and Spark 4.1.4/Scala 
2.13/JDK 17; current-master integration CI is tracked on the PR.
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR. The proposed implementation is #11322.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to