mvanhorn opened a new pull request, #11102: URL: https://github.com/apache/paimon/pull/11102
### Purpose If a job stops after one commit identifier has published its first snapshot and before the compaction snapshot for that same identifier, restore drops the compaction. The next compaction then deletes an output file that is absent from the table, conflict detection aborts the commit, and the job restarts. Each table left in that window costs another restart, and the changelog the dropped compaction would have published is missing. One commit identifier is published as two snapshots, first APPEND or OVERWRITE and then COMPACT. On restore a committable is kept only when its identifier is greater than the latest snapshot identifier for this commit user, so an identifier whose first snapshot already exists is dropped together with its still unpublished compact increments. The writer of the next identifier already had those compaction outputs in memory and emits deletes for them, which conflict detection rejects because the files were never added. When the committable identifier equals that latest snapshot and the snapshot kind is APPEND or OVERWRITE, filterCommitted adds a separate committable that copies the watermark and properties and keeps only the non-empty compact increments, each with an empty data increment. CompactRecoveryCommittable marks that copy so commit writes no further append snapshot and skips overwrite, and the replay publishes one COMPACT snapshot even when an empty append would otherwise be written. Before replay, every file from the compact-after and changelog metas and every new index file is resolved through CompactedChangelogPathResolver and checked for existence; a missing output skips the whole identifier, and an I/O error from the existence check fails recovery as UncheckedIOException. Fixes #11071 ### Tests mvn -B -ntp -pl paimon-core -am test -Dtest=FileStoreCommitTest,TableCommitTest -Dsurefire.failIfNoSpecifiedTests=false finished TableCommitTest with 30 tests, 0 failures, 0 errors, and 0 skipped, then timed out at 300s. No test was run with the paimon FileStoreCommitImpl change reverted, because no test was named as covering the reported bug, so none is shown to catch this bug. The changed test paimon-core/src/test/java/org/apache/paimon/operation/FileStoreCommitTest.java did not run here: no test runner for it was available. The changed test paimon-core/src/test/java/org/apache/paimon/table/sink/TableCommitTest.java did not run here: no test runner for it was available. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
