Akash3121 commented on code in PR #11102:
URL: https://github.com/apache/paimon/pull/11102#discussion_r4545607227
##########
paimon-core/src/main/java/org/apache/paimon/operation/FileStoreCommitImpl.java:
##########
@@ -324,6 +329,15 @@ public List<ManifestCommittable>
filterCommitted(List<ManifestCommittable> commi
result.add(committable);
} else {
commitCallbacks.forEach(callback ->
callback.retry(committable));
+ Snapshot snapshot = latestSnapshot.get();
+ if (committable.identifier() == snapshot.commitIdentifier()
+ && (snapshot.commitKind() == CommitKind.APPEND
Review Comment:
Do not replay a compact increment whose inputs have been concurrently removed
This branch recovers the COMPACT whenever its output files exist, but that
is not sufficient to make the increment replayable. A concurrent compaction may
have already deleted one of its `compactBefore` files. In that case conflict
detection rejects this stale delete; after restart `filterCommitted`
reconstructs the same recovery committable again, so the job can never
progress. The existing
`PrimaryKeyFileStoreTableITCase.testBatchJobWithConflictAndRestart`
demonstrates this in the failed Flink 2 CI job and in the local run: all 10
restart attempts end in `File deletion conflicts detected`.
Please validate the compact inputs against the current snapshot and
discard/rebase an invalid recovery (with coverage for this
concurrent-compaction case) instead of unconditionally replaying it based only
on output existence + adding a test case might help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]