taoran92 opened a new pull request, #10448: URL: https://github.com/apache/paimon/pull/10448
### Purpose Closes #10447. Repeated append-only compaction before `prepareCommit` retains intermediate files in both `compactBefore` and `compactAfter`. For example, `A + B + C -> X` followed by `X + D + E -> Y` emits both DELETE and ADD entries for `X`, which can fail manifest merging. Track pending compaction files by file name and remove outputs consumed by a later compaction from the pending results. Delete those intermediate data files and their extra files, while preserving files already handed to a committer by an earlier `prepareCommit`. ### Tests Added `AppendOnlyWriterTest#testRepeatedCompaction`, covering repeated compaction with and without an intervening `prepareCommit`. It checks the compaction increments, intermediate data and extra-file cleanup, preservation of files handed to a committer, and manifest merging to the final output. Passed locally: 2 parameterized cases, 0 failures or errors. The command also passed the module's Checkstyle, Spotless, and Enforcer checks without `fast-build`: ```bash mvn -pl paimon-core -DwildcardSuites=none '-Dtest=AppendOnlyWriterTest#testRepeatedCompaction' test ``` `git diff --check apache/master...HEAD` also passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
