Akash3121 commented on code in PR #11102:
URL: https://github.com/apache/paimon/pull/11102#discussion_r4545607227


##########
paimon-core/src/main/java/org/apache/paimon/operation/FileStoreCommitImpl.java:
##########
@@ -324,6 +329,15 @@ public List<ManifestCommittable> 
filterCommitted(List<ManifestCommittable> commi
                     result.add(committable);
                 } else {
                     commitCallbacks.forEach(callback -> 
callback.retry(committable));
+                    Snapshot snapshot = latestSnapshot.get();
+                    if (committable.identifier() == snapshot.commitIdentifier()
+                            && (snapshot.commitKind() == CommitKind.APPEND

Review Comment:
   Do not replay a compact increment whose inputs have been concurrently removed
    
   This branch recovers the COMPACT whenever its output files exist, but that 
is not sufficient to make the increment replayable. A concurrent compaction may 
have already deleted one of its `compactBefore` files. In that case conflict 
detection rejects this stale delete; after restart `filterCommitted` 
reconstructs the same recovery committable again, so the job can never 
progress. The existing 
`PrimaryKeyFileStoreTableITCase.testBatchJobWithConflictAndRestart` 
demonstrates this in the failed Flink 2 CI job and in the local run: all 10 
restart attempts end in `File deletion conflicts detected`. 
   
   Please validate the compact inputs against the current snapshot and 
discard/rebase an invalid recovery (with coverage for this 
concurrent-compaction case) instead of unconditionally replaying it based only 
on output existence + adding a test case might help.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to