JustinBinber opened a new pull request, #11156: URL: https://github.com/apache/paimon/pull/11156
Draft: the fixed-resource capacity case, exact correctness checks, normal builds, focused regressions, independent source review and balanced32-file series are complete. The preregistered65536-file performance series and offline resource summaries are still in progress. This draft starts upstream CI in parallel; it is not a claim of final performance acceptance. ### Purpose Spark Data Evolution MERGE can exhaust the Driver heap when the table contains many small column files, even when the row count is modest. Metadata grows during scan planning, task-result decoding and commit preparation; reducing one collection alone can move the failure to a later stage. Add one opt-in table option, `data-evolution.metadata-memory-optimization.mode` (`off`, `lightweight`, `streaming`; default `off`). Lightweight mode reduces eager decoding, copies and overlapping object lifetimes. Streaming mode additionally sorts/replays file metadata and plans complete RowID dependency groups incrementally, then uses compact summaries and external identity reconciliation in the production commit path. Partial-write tasks produce commit messages on demand. The change preserves a single atomic Snapshot and retains compatibility paths for deletion vectors and callbacks. The scope is the planning-to-commit chain required by the reproduced Data Evolution MERGE failure. This does not establish that the separate production SELECT * LIMIT 20 incident is fixed, or that execution has constant memory. Task arrays, complete result bytes, the largest message/dependency group, RowID summaries and compatibility/diagnostic paths remain boundaries. Local replay/sort I/O and additional decoding are measured costs, not assumed speedups. ### Tests Fixed upstream comparison commit: b7cfb397de334d56b8c3eaeb60a2cfa9c9cab627. Published candidate: 42540b17fd0cf7b8e5ff05efb598b88abfd76ecd, with exactly the same Git tree as frozen f17caf4cc27eaa87927fe0fb28385462aecb373d. Runtime artifacts are bound to normal Maven builds; Spark3 applicable inputs are unchanged by the final Spark4-only formatting commit. The warm Maven cache was shared; current reactor/module outputs and runtime class origins were checked separately. - Current Core risk regressions: 50 passed. Current Spark3 SQL/metadata/concurrency regressions: 23 passed. - Normal Spark4.0/4.1 package passed. Spark4.1: 11 tests through normal Maven. Spark4.0: 11 through direct ScalaTest on the normal reactor classpath, with actual Spark4.0.5 and current packaged Paimon classes. This is not a default UT40 Maven-test claim. Spark4.2 is not locally covered. - Current 32-file MERGE prerequisite: passed with Driver 1 GiB and two 512 MiB Executors; five exact result/RowID checks passed, and one Snapshot was published. Its separate oracle uses Driver 2 GiB and two 512 MiB Executors. This pilot is excluded from timing samples. - Fixed-b7 clean upstream, 524288 real files, Driver 1 GiB/two 512 MiB Executors: actual Driver heap OOM during scan planning; 35.397 s process wall time, no new Snapshot (still 512), source and seed unchanged. This is a capacity failure observation, not a latency baseline. The same-input candidate MERGE exited successfully in 740.262 s SQL time and published Snapshot513 with unchanged measured heaps. The separate baseline oracle (4g/1g/1g) passed all five exact full-result/RowID checks (1153433 rows); source/seed and all prior Snapshots are unchanged. Both controller phases exited0 without disk/memory reserve violations. - Successful-case32-file performance: all four balanced Williams blocks (16 measured MERGEs and separate five-check oracles) passed. SQL median seconds: upstream7.950692, off7.737670, lightweight7.800494, streaming8.083136. Streaming is +1.67% by ratio of medians; all four raw samples/ranges and paired comparisons are retained, with no p95 or broad speedup claim. The65536-file campaign remains PENDING. No speedup is calculated against an OOM. The benchmark environment is one 4-core/16-GiB Linux machine with separate Spark processes; it is not a multiple-worker-node cluster benchmark. The 524288-file correctness oracle uses a separately disclosed larger heap. The [source-only reproduction kit and recorded results](https://github.com/JustinBinber/paimon/tree/5acdda1a7531e40e5171b1e749bba1a23bcaa02b) reuse the original helper/controller and disclose fixture origin, resource settings, exact SQL and remaining limits. The assembled kit has not been rerun as a fresh-host package. The remaining65536 balanced timing blocks and CPU/GC/Shuffle/process I/O summaries will be added before this draft is marked ready for review. The fork has GitHub-controlled Actions disabling: the workflow is active and the repository Actions policy is enabled, but dispatch returns `Actions has been disabled for this repository.` No green fork-CI claim is made; upstream pull-request checks will be tracked. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
