JustinBinber opened a new pull request, #11322:
URL: https://github.com/apache/paimon/pull/11322

   ### Purpose
   
   The shared Data Evolution MERGE join currently caches source columns needed 
only by INSERT along with the matching and UPDATE fields. For a wide source 
this sends otherwise unused payload through the shared shuffle and executor 
disk cache.
   
   This adds a default-off option, 
`data-evolution.merge-into.join-reuse.narrow-source.enabled`, used with the 
existing join-reuse option. The shared join retains matching and UPDATE 
dependencies plus presence markers. INSERT obtains unmatched source keys from 
that result and uses a null-safe membership join against a reread of the pinned 
source. This remains a source scan, not a positional read. DE-source RowID 
reads belong to a separate change.
   
   The reread is restricted to a single INT/BIGINT equality key and safe 
analyzed Paimon append-table source plans whose snapshot can be pinned. The 
same source snapshot and attribute ExprIds are preserved. Unsupported 
sources/plans, self reads, composite branch readers, missing snapshot metadata 
and incompatible schema-only evolution retain the existing path. Users do not 
have to assert a `stableSource` promise. NULL keys and duplicate unmatched rows 
retain multiset semantics, and cached matches are released on failure.
   
   Example opt-in:
   
   ```sql
   ALTER TABLE target SET TBLPROPERTIES (
     'data-evolution.merge-into.join-reuse' = 'true',
     'data-evolution.merge-into.join-reuse.narrow-source.enabled' = 'true'
   );
   ```
   
   Source: `4964431e3d56e161dd29cf87fe3920dd6881a6c2`. [Synthetic reproduction 
sources, exact SQL generators, nine sample metrics, validation summaries and 
limitations](https://github.com/JustinBinber/paimon/tree/0c5bd5e93b74dad1160813daa4f45d86c2db8756/benchmarks/merge-narrow-source).
 The reproduction branch is separate from this production PR diff.
   
   ### Tests
   
   Validation: 17 relevant tests on Spark 3.5.9 / Scala 2.12 / JDK 8 and 17 on 
Spark 4.1.4 / Scala 2.13 / JDK 17. The Spark 4.0 mirror compiled and passed 
module validation; no Spark 4.0 runtime result is claimed. Tests cover snapshot 
recomputation/time travel, schema-only changes, retained tags, branches and 
composite readers, empty sources, unsupported plans, NULL equality, false 
UPDATE conditions, duplicate unmatched rows and failure cleanup.
   
   ### Performance observations
   
   Bounded benchmark: one 4-core/16-GiB physical host, two one-core/1-GiB 
Executor JVMs, 4-GiB Driver, Spark 3.5.9 with AQE/coalescing enabled. Target: 
32,768 two-row DE append files, 259 columns; UPDATE changes 128 columns. 
Ordinary append source: 39,321 rows, with a real 8-KiB logical INSERT-only 
field where applicable. Every arm used a fresh target clone/JVM, identical 
source dataset and settings, complete OS input warm-up, full SQL timing, and 
independent complete-result/RowID/snapshot/layout validation.
   
   | Scenario, one sample per arm | No sharing | Existing sharing | Narrow 
enabled |
   |---|---:|---:|---:|
   | 90% INSERT | 77.85 s | 76.95 s | 72.63 s |
   | All INSERT | 52.73 s | 53.37 s | 51.58 s |
   | 90% INSERT without an INSERT-only wide field | 78.41 s | 71.84 s | 73.42 s 
|
   
   These are screening observations, not significant or stable speedup claims. 
All-INSERT used the fallback path with zero shared-cache/shuffle bytes. The 
no-wide-field control had identical shuffle/cache bytes for existing/narrow 
sharing, but the narrow-enabled timing was 2.21% higher in this single 
observation. No default enablement or automatic INSERT-ratio threshold is 
proposed.
   
   For the 90%-INSERT wide-field case, existing/narrow sharing produced 
316.79/10.77 MB shuffle, 312.20/18.78 MB peak shared disk cache, and 
1,300.89/381.79 MB sampled process writes. **Sampled process-tree RSS was 
higher: 3.78/4.80 GiB (+27%); the no-sharing reference was 4.66 GiB.** The 
full-SQL observation improved 5.61% relative to existing sharing. Source FileIO 
returned approximately 164 MB in both paths; this is not physical disk/network 
traffic. Sampled process counter deltas are interior lower bounds, not complete 
process totals.
   
   All nine SQLs and independent validations passed without timed task failure 
or disk spill. Two short warm INSERT-range SELECT probes were also retained, 
including an all-INSERT narrow-enabled observation 4.28% slower than existing 
sharing; they do not prove general query non-regression. The higher memory 
observation and these limited samples are explicit tradeoffs of the opt-in path.
   
   Negative/lifecycle evidence is retained: an all-INSERT pre-launch 
disk-budget rejection (no SQL executed), old build cleanup before continuing 
the missing phase. Successful SQL samples were never replayed or selectively 
replaced. The 5-GiB disk and 3.2-GiB available-memory floors were not lowered.
   
   Merge preview against current master 
`48da48ae2e7ba1f29fb380218481d3555c62f1c0` has no conflicts. The retained 
runtime tests bind to the unchanged feature source; current-master integration 
still requires CI acceptance.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to