mvanhorn opened a new pull request, #11102:
URL: https://github.com/apache/paimon/pull/11102

   ### Purpose
   
   If a job stops after one commit identifier has published its first snapshot 
and before the compaction snapshot for that same identifier, restore drops the 
compaction. The next compaction then deletes an output file that is absent from 
the table, conflict detection aborts the commit, and the job restarts. Each 
table left in that window costs another restart, and the changelog the dropped 
compaction would have published is missing. One commit identifier is published 
as two snapshots, first APPEND or OVERWRITE and then COMPACT.
   
   On restore a committable is kept only when its identifier is greater than 
the latest snapshot identifier for this commit user, so an identifier whose 
first snapshot already exists is dropped together with its still unpublished 
compact increments. The writer of the next identifier already had those 
compaction outputs in memory and emits deletes for them, which conflict 
detection rejects because the files were never added.
   
   When the committable identifier equals that latest snapshot and the snapshot 
kind is APPEND or OVERWRITE, filterCommitted adds a separate committable that 
copies the watermark and properties and keeps only the non-empty compact 
increments, each with an empty data increment. CompactRecoveryCommittable marks 
that copy so commit writes no further append snapshot and skips overwrite, and 
the replay publishes one COMPACT snapshot even when an empty append would 
otherwise be written.
   
   Before replay, every file from the compact-after and changelog metas and 
every new index file is resolved through CompactedChangelogPathResolver and 
checked for existence; a missing output skips the whole identifier, and an I/O 
error from the existence check fails recovery as UncheckedIOException.
   
   Fixes #11071
   
   ### Tests
   
   mvn -B -ntp -pl paimon-core -am test 
-Dtest=FileStoreCommitTest,TableCommitTest 
-Dsurefire.failIfNoSpecifiedTests=false finished TableCommitTest with 30 tests, 
0 failures, 0 errors, and 0 skipped, then timed out at 300s.
   No test was run with the paimon FileStoreCommitImpl change reverted, because 
no test was named as covering the reported bug, so none is shown to catch this 
bug.
   The changed test 
paimon-core/src/test/java/org/apache/paimon/operation/FileStoreCommitTest.java 
did not run here: no test runner for it was available.
   The changed test 
paimon-core/src/test/java/org/apache/paimon/table/sink/TableCommitTest.java did 
not run here: no test runner for it was available.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to