JingsongLi commented on PR #9177: URL: https://github.com/apache/paimon/pull/9177#issuecomment-5262744638
@steFaiz Thanks, this is a good question. The main targets are not only deletion-vector materialization. There are two common Data Evolution compaction scenarios: 1. **Only part of the existing files/ranges are updated.** The projected first pass identifies the exact affected row-ID ranges, and the second pass loads full metadata only for those ranges. This reduces both the retained heap and the amount of full metadata read. 2. **Newly written files need small-file compaction.** Only the row-ID ranges where the new files form a compaction candidate (plus the normal/dedicated-file closure required for correctness) are loaded with full metadata. Previously compacted historical ranges are not retained as full `DataFileMeta` objects. If adding/backfilling a column really touches almost every row range, then the second pass may indeed read nearly all metadata, so the two-phase design does not reduce total manifest I/O in that worst case and adds a lightweight projected pass. However, it still substantially reduces **peak heap**: the first pass retains compact primitive candidate metadata, while full file metadata is materialized and committed one range batch at a time instead of for the whole table. The 100k target is intentionally soft because one connected logical range cannot be split safely. So the primary guarantee is a bounded working set / avoiding OOM. In the common partial-update and newly-added-small-file cases, it also avoids reading most full metadata. DV materialization is another strong beneficiary, but not the only target. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
