JTaky opened a new issue, #11071: URL: https://github.com/apache/paimon/issues/11071
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version 2.0.0 ### Compute Engine Flink 2.2 (multi-table CDC sink) ### Minimal reproduce step FileStoreCommitImpl.commit() writes the changes of one commit identifier as two separate snapshots: APPEND(N) first, then COMPACT(N). On restore, filterCommitted() drops every committable whose identifier is <= the latest snapshot of this commit user, based on the identifier alone: ``` if (committable.identifier() > latestSnapshot.get().commitIdentifier()) { result.add(committable); } ``` If the job dies after APPEND(N) and before COMPACT(N): 1. The restored state contains committables N and N+1, which were checkpointed but not yet committed. 2. filterCommitted sees that the latest snapshot has identifier N and drops committable N entirely, including its compact changes. The compaction output file X is never added to the table. 3. Committable N+1 was produced by a writer whose in-memory file list already contained X, so its compaction deletes X. 4. COMPACT(N+1) fails conflict detection with Trying to delete file X ... which is not previously added, and the job restarts. 5. On the next restore, N+1 is half-committed in the same way and gets dropped, so the table heals. With a multi-table sink, every table that was half-committed at the moment of failure costs one more restart. Unit test (add to TableCommitTest) that fails on 2.0.0: ``` @Test public void testRecoverCompactOfHalfCommittedIdentifier() throws Exception { FileStoreTable table = createPrimaryKeyTable(); // pk "k", bucket = 1 String user = UUID.randomUUID().toString(); TableWriteImpl<?> write = table.newWrite(user); TableCommitImpl commit = table.newCommit(user); write.write(GenericRow.of(1, 1)); commit.commit(1, write.prepareCommit(false, 1)); write.write(GenericRow.of(2, 2)); write.compact(BinaryRow.EMPTY_ROW, 0, true); List<CommitMessage> messages2 = write.prepareCommit(true, 2); // the job dies between the append and the compact snapshot of identifier 2 commit.commit(2, appendOnly(messages2)); // the writer keeps compacting on top of identifier 2's compaction output write.write(GenericRow.of(3, 3)); write.compact(BinaryRow.EMPTY_ROW, 0, true); List<CommitMessage> messages3 = write.prepareCommit(true, 3); Map<Long, List<CommitMessage>> restored = new HashMap<>(); restored.put(2L, messages2); restored.put(3L, messages3); table.newCommit(user).filterAndCommit(restored); // throws } ``` appendOnly(...) copies each CommitMessageImpl with CompactIncrement.emptyIncrement(). The test throws: > IllegalStateException: Trying to delete file data-...-0.parquet for table ... which is not previously added. > at ConflictDetection.checkDeleteInEntries(ConflictDetection.java:429) > at FileStoreCommitImpl.commit(FileStoreCommitImpl.java:381) > at TableCommitImpl.filterAndCommitMultiple(TableCommitImpl.java:331) ### What doesn't meet your expectations? Recovery should be idempotent per snapshot, not per identifier. A failure between the two snapshots of one identifier should not lose the compaction, or make the next identifier's compaction fail. Job impact: frequent compaction keeps the time window between APPEND and COMPACT open almost all the time. A cancellation in that window (for example a savepoint timeout) put the job into a restart cascade: 27 restarts, about 35 minutes, one new table per restart. The changelog produced by a dropped compaction (lookup / full-compaction producers) is also lost silently. ### Anything else? Proposed fix, in filterCommitted: when committable.identifier() == latest.commitIdentifier() and latest.commitKind() != COMPACT, keep a copy of the committable that contains only its compact increments, so the compaction is recommitted. If any compaction output file no longer exists (for example it was removed as an orphan), fall back to today's behaviour and skip it, so recovery doesn't get stuck in checkFilesExistence. With the fix, the test above produces 1:APPEND, 2:APPEND, 2:COMPACT, 3:APPEND, 3:COMPACT and all rows are readable. The alternative is to write APPEND and COMPACT as a single snapshot. That removes the window entirely, but it changes snapshot semantics for streaming readers. ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
