NestDream opened a new issue, #9958: URL: https://github.com/apache/paimon/issues/9958
### Search before asking - [x] I searched in the issues and found nothing similar. ### Paimon version master (8b7fbffcc). The line is the same in release-1.2 through 2.1. ### Compute Engine Flink 1.20.1, `GlobalFullCompactionSinkWrite` (used when `changelog-producer=full-compaction`). ### Minimal reproduce step 1. Create a primary-key table with `'changelog-producer' = 'full-compaction'` and `'full-compaction.delta-commits' = '10'`, stream into it with a 500 ms checkpoint interval (a datagen source is enough). 2. Set `org.apache.paimon.flink.sink.GlobalFullCompactionSinkWrite` to DEBUG. 3. Watch the line `Found full compaction snapshot #... with identifier ...` after the first full compaction. ### What doesn't meet your expectations? The DEBUG line is printed on every checkpoint after the first full compaction (50 times in 60 checkpoints), not once per full compaction. Each print is one walk of `checkSuccessfulFullCompaction` over the snapshot files from latest down to the COMPACT snapshot. Cause: when the COMPACT snapshot of a submitted identifier is found, the code clears `writtenBuckets.headMap(commitIdentifier, true)` (inclusive) but `commitIdentifiersToCheck.headSet(commitIdentifier)` (exclusive), so the identifier it just confirmed stays in the set and every later `prepareCommit` walks the snapshots again until the next full compaction replaces it. Measured on a local MiniCluster with a counting `LocalFileIO` (60 checkpoints, six runs): 50 walks, about 280 snapshot file reads plus 100 hint file reads from this check; with `headSet(commitIdentifier, true)` it is 5 walks, 5 snapshot reads and 10 hint reads. Every writer subtask does this, so the extra reads per checkpoint scale with parallelism and `full-compaction.delta-commits`, each read being a GET on an object store. ### Anything else? One-line fix: `commitIdentifiersToCheck.headSet(commitIdentifier, true).clear();`, matching the `headMap` call on the line above and the other `headSet`/`headMap` uses in paimon-flink. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
