JustinBinber opened a new pull request, #11156:
URL: https://github.com/apache/paimon/pull/11156

   Draft: the fixed-resource capacity case, exact correctness checks, normal 
builds, focused regressions, independent source review and balanced32-file 
series are complete. The preregistered65536-file performance series and offline 
resource summaries are still in progress. This draft starts upstream CI in 
parallel; it is not a claim of final performance acceptance.
   
   ### Purpose
   
   Spark Data Evolution MERGE can exhaust the Driver heap when the table 
contains many small column files, even when the row count is modest. Metadata 
grows during scan planning, task-result decoding and commit preparation; 
reducing one collection alone can move the failure to a later stage.
   
   Add one opt-in table option, 
`data-evolution.metadata-memory-optimization.mode` (`off`, `lightweight`, 
`streaming`; default `off`). Lightweight mode reduces eager decoding, copies 
and overlapping object lifetimes. Streaming mode additionally sorts/replays 
file metadata and plans complete RowID dependency groups incrementally, then 
uses compact summaries and external identity reconciliation in the production 
commit path. Partial-write tasks produce commit messages on demand. The change 
preserves a single atomic Snapshot and retains compatibility paths for deletion 
vectors and callbacks.
   
   The scope is the planning-to-commit chain required by the reproduced Data 
Evolution MERGE failure. This does not establish that the separate production 
SELECT * LIMIT 20 incident is fixed, or that execution has constant memory. 
Task arrays, complete result bytes, the largest message/dependency group, RowID 
summaries and compatibility/diagnostic paths remain boundaries. Local 
replay/sort I/O and additional decoding are measured costs, not assumed 
speedups.
   
   ### Tests
   
   Fixed upstream comparison commit: b7cfb397de334d56b8c3eaeb60a2cfa9c9cab627. 
Published candidate: 42540b17fd0cf7b8e5ff05efb598b88abfd76ecd, with exactly the 
same Git tree as frozen f17caf4cc27eaa87927fe0fb28385462aecb373d. Runtime 
artifacts are bound to normal Maven builds; Spark3 applicable inputs are 
unchanged by the final Spark4-only formatting commit. The warm Maven cache was 
shared; current reactor/module outputs and runtime class origins were checked 
separately.
   
   - Current Core risk regressions: 50 passed. Current Spark3 
SQL/metadata/concurrency regressions: 23 passed.
   - Normal Spark4.0/4.1 package passed. Spark4.1: 11 tests through normal 
Maven. Spark4.0: 11 through direct ScalaTest on the normal reactor classpath, 
with actual Spark4.0.5 and current packaged Paimon classes. This is not a 
default UT40 Maven-test claim. Spark4.2 is not locally covered.
   - Current 32-file MERGE prerequisite: passed with Driver 1 GiB and two 512 
MiB Executors; five exact result/RowID checks passed, and one Snapshot was 
published. Its separate oracle uses Driver 2 GiB and two 512 MiB Executors. 
This pilot is excluded from timing samples.
   - Fixed-b7 clean upstream, 524288 real files, Driver 1 GiB/two 512 MiB 
Executors: actual Driver heap OOM during scan planning; 35.397 s process wall 
time, no new Snapshot (still 512), source and seed unchanged. This is a 
capacity failure observation, not a latency baseline. The same-input candidate 
MERGE exited successfully in 740.262 s SQL time and published Snapshot513 with 
unchanged measured heaps. The separate baseline oracle (4g/1g/1g) passed all 
five exact full-result/RowID checks (1153433 rows); source/seed and all prior 
Snapshots are unchanged. Both controller phases exited0 without disk/memory 
reserve violations.
   - Successful-case32-file performance: all four balanced Williams blocks (16 
measured MERGEs and separate five-check oracles) passed. SQL median seconds: 
upstream7.950692, off7.737670, lightweight7.800494, streaming8.083136. 
Streaming is +1.67% by ratio of medians; all four raw samples/ranges and paired 
comparisons are retained, with no p95 or broad speedup claim. The65536-file 
campaign remains PENDING. No speedup is calculated against an OOM.
   
   The benchmark environment is one 4-core/16-GiB Linux machine with separate 
Spark processes; it is not a multiple-worker-node cluster benchmark. The 
524288-file correctness oracle uses a separately disclosed larger heap. The 
[source-only reproduction kit and recorded 
results](https://github.com/JustinBinber/paimon/tree/5acdda1a7531e40e5171b1e749bba1a23bcaa02b)
 reuse the original helper/controller and disclose fixture origin, resource 
settings, exact SQL and remaining limits. The assembled kit has not been rerun 
as a fresh-host package. The remaining65536 balanced timing blocks and 
CPU/GC/Shuffle/process I/O summaries will be added before this draft is marked 
ready for review.
   
   The fork has GitHub-controlled Actions disabling: the workflow is active and 
the repository Actions policy is enabled, but dispatch returns `Actions has 
been disabled for this repository.` No green fork-CI claim is made; upstream 
pull-request checks will be tracked.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to