JustinBinber opened a new pull request, #11322:
URL: https://github.com/apache/paimon/pull/11322
### Purpose
The shared Data Evolution MERGE join currently caches source columns needed
only by INSERT along with the matching and UPDATE fields. For a wide source
this sends otherwise unused payload through the shared shuffle and executor
disk cache.
This adds a default-off option,
`data-evolution.merge-into.join-reuse.narrow-source.enabled`, used with the
existing join-reuse option. The shared join retains matching and UPDATE
dependencies plus presence markers. INSERT obtains unmatched source keys from
that result and uses a null-safe membership join against a reread of the pinned
source. This remains a source scan, not a positional read. DE-source RowID
reads belong to a separate change.
The reread is restricted to a single INT/BIGINT equality key and safe
analyzed Paimon append-table source plans whose snapshot can be pinned. The
same source snapshot and attribute ExprIds are preserved. Unsupported
sources/plans, self reads, composite branch readers, missing snapshot metadata
and incompatible schema-only evolution retain the existing path. Users do not
have to assert a `stableSource` promise. NULL keys and duplicate unmatched rows
retain multiset semantics, and cached matches are released on failure.
Example opt-in:
```sql
ALTER TABLE target SET TBLPROPERTIES (
'data-evolution.merge-into.join-reuse' = 'true',
'data-evolution.merge-into.join-reuse.narrow-source.enabled' = 'true'
);
```
Source: `4964431e3d56e161dd29cf87fe3920dd6881a6c2`. [Synthetic reproduction
sources, exact SQL generators, nine sample metrics, validation summaries and
limitations](https://github.com/JustinBinber/paimon/tree/0c5bd5e93b74dad1160813daa4f45d86c2db8756/benchmarks/merge-narrow-source).
The reproduction branch is separate from this production PR diff.
### Tests
Validation: 17 relevant tests on Spark 3.5.9 / Scala 2.12 / JDK 8 and 17 on
Spark 4.1.4 / Scala 2.13 / JDK 17. The Spark 4.0 mirror compiled and passed
module validation; no Spark 4.0 runtime result is claimed. Tests cover snapshot
recomputation/time travel, schema-only changes, retained tags, branches and
composite readers, empty sources, unsupported plans, NULL equality, false
UPDATE conditions, duplicate unmatched rows and failure cleanup.
### Performance observations
Bounded benchmark: one 4-core/16-GiB physical host, two one-core/1-GiB
Executor JVMs, 4-GiB Driver, Spark 3.5.9 with AQE/coalescing enabled. Target:
32,768 two-row DE append files, 259 columns; UPDATE changes 128 columns.
Ordinary append source: 39,321 rows, with a real 8-KiB logical INSERT-only
field where applicable. Every arm used a fresh target clone/JVM, identical
source dataset and settings, complete OS input warm-up, full SQL timing, and
independent complete-result/RowID/snapshot/layout validation.
| Scenario, one sample per arm | No sharing | Existing sharing | Narrow
enabled |
|---|---:|---:|---:|
| 90% INSERT | 77.85 s | 76.95 s | 72.63 s |
| All INSERT | 52.73 s | 53.37 s | 51.58 s |
| 90% INSERT without an INSERT-only wide field | 78.41 s | 71.84 s | 73.42 s
|
These are screening observations, not significant or stable speedup claims.
All-INSERT used the fallback path with zero shared-cache/shuffle bytes. The
no-wide-field control had identical shuffle/cache bytes for existing/narrow
sharing, but the narrow-enabled timing was 2.21% higher in this single
observation. No default enablement or automatic INSERT-ratio threshold is
proposed.
For the 90%-INSERT wide-field case, existing/narrow sharing produced
316.79/10.77 MB shuffle, 312.20/18.78 MB peak shared disk cache, and
1,300.89/381.79 MB sampled process writes. **Sampled process-tree RSS was
higher: 3.78/4.80 GiB (+27%); the no-sharing reference was 4.66 GiB.** The
full-SQL observation improved 5.61% relative to existing sharing. Source FileIO
returned approximately 164 MB in both paths; this is not physical disk/network
traffic. Sampled process counter deltas are interior lower bounds, not complete
process totals.
All nine SQLs and independent validations passed without timed task failure
or disk spill. Two short warm INSERT-range SELECT probes were also retained,
including an all-INSERT narrow-enabled observation 4.28% slower than existing
sharing; they do not prove general query non-regression. The higher memory
observation and these limited samples are explicit tradeoffs of the opt-in path.
Negative/lifecycle evidence is retained: an all-INSERT pre-launch
disk-budget rejection (no SQL executed), old build cleanup before continuing
the missing phase. Successful SQL samples were never replayed or selectively
replaced. The 5-GiB disk and 3.2-GiB available-memory floors were not lowered.
Merge preview against current master
`48da48ae2e7ba1f29fb380218481d3555c62f1c0` has no conflicts. The retained
runtime tests bind to the unchanged feature source; current-master integration
still requires CI acceptance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]