leaves12138 opened a new pull request, #9213: URL: https://github.com/apache/paimon/pull/9213
### Purpose Optimize ordinary (non-sorted) manifest full and minor compaction without materializing every manifest entry. The legacy merger decodes complete `ManifestEntry` objects, builds an identifier-to-entry map, and rewrites every surviving entry. On large data-evolution tables this creates high allocation pressure and makes manifest merging a major part of commit latency. This PR introduces a block-aware ordinary merger which: - reads DELETE entries with a minimal projection; - uses primitive RowID and identifier indexes to reject unaffected ADD entries cheaply; - copies an unchanged Avro block, or an entire compatible add-only manifest, without decoding and re-encoding every record; - decodes and filters individual entries only for blocks that may contain a matching DELETE or whose encoded schema is incompatible; - keeps planning memory bounded to the manifests handled by the configured read batch; - retains the previous implementation as `ManifestFileLegacyMerger`, selectable with `manifest.merge-optimize.enabled=false`. This PR only changes ordinary manifest full/minor compaction. The manifest sort-compaction branch and `ManifestFileSorter` are unchanged. ### Correctness validation Benchmarks used real metadata from `dp_lowprec_clip_images`. Each mode was warmed up once and then measured without an `-Xmx` limit. Output manifests were scanned and compared with an order-independent content fingerprint containing entry kind, file identifier and first RowID. Every legacy/optimized pair produced the same entry count, ADD/DELETE counts, and content fingerprint. #### Minor compaction | Snapshot | Legacy | Block-aware | Speedup | Peak RSS legacy | Peak RSS block-aware | RSS reduction | | --- | ---: | ---: | ---: | ---: | ---: | ---: | | 8795 | 6.524 s | 0.053 s | **124.3x** | 10.30 GiB | 0.94 GiB | **90.9%** | | 8800 | 15.685 s | 0.104 s | **150.3x** | 11.91 GiB | 0.87 GiB | **92.7%** | | 8808 | 4.340 s | 0.040 s | **108.6x** | 5.40 GiB | 1.26 GiB | **76.6%** | #### Full compaction | Snapshot | Legacy | Block-aware | Speedup | Peak RSS legacy | Peak RSS block-aware | RSS reduction | | --- | ---: | ---: | ---: | ---: | ---: | ---: | | 8795 | 34.214 s | 5.963 s | **5.74x** | 16.26 GiB | 1.54 GiB | **90.6%** | | 8808 | 38.332 s | 6.653 s | **5.76x** | 16.24 GiB | 1.42 GiB | **91.2%** | | 8816 | 39.123 s | 6.310 s | **6.20x** | 16.26 GiB | 1.29 GiB | **92.1%** | Peak RSS covers the whole benchmark JVM, including warm-up and output verification, so it is intentionally more conservative than measuring the merge allocation alone. Different output byte sizes are expected because raw-block reuse changes Avro block boundaries/compression layout; logical contents matched in all six comparisons. ### Tests - Linux x86_64 targeted regression on dev2: 19 tests, 0 failures. - Added full/minor tests for add-only raw-block reuse, unaffected blocks around DELETE entries, non-RowID identifier filtering, fallback to the legacy merger, bounded manifest scans, encoded-record buffer flushing, aggregate-stat preservation, and binary-layout compatibility. - `~/bin/check` - `git diff --check` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
