JTaky opened a new issue, #11071:
URL: https://github.com/apache/paimon/issues/11071

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon/issues) 
and found nothing similar.
   
   
   ### Paimon version
   
   2.0.0
   
   ### Compute Engine
   
   Flink 2.2 (multi-table CDC sink)
   
   ### Minimal reproduce step
   
   FileStoreCommitImpl.commit() writes the changes of one commit identifier as 
two separate snapshots: APPEND(N) first, then COMPACT(N). On restore, 
filterCommitted() drops every committable whose identifier is <= the latest 
snapshot of this commit user, based on the identifier alone:
   ```
   if (committable.identifier() > latestSnapshot.get().commitIdentifier()) {
       result.add(committable);
   }
   ```
   
   If the job dies after APPEND(N) and before COMPACT(N):
   1. The restored state contains committables N and N+1, which were 
checkpointed but not yet committed.
   2. filterCommitted sees that the latest snapshot has identifier N and drops 
committable N entirely, including its compact changes. The compaction
      output file X is never added to the table.
   3. Committable N+1 was produced by a writer whose in-memory file list 
already contained X, so its compaction deletes X.
   4. COMPACT(N+1) fails conflict detection with Trying to delete file X ... 
which is not previously added, and the job restarts.
   5. On the next restore, N+1 is half-committed in the same way and gets 
dropped, so the table heals. With a multi-table sink, every table that was
      half-committed at the moment of failure costs one more restart.           
                                                                     
   Unit test (add to TableCommitTest) that fails on 2.0.0:
   
   ```
   @Test
   public void testRecoverCompactOfHalfCommittedIdentifier() throws Exception {
       FileStoreTable table = createPrimaryKeyTable(); // pk "k", bucket = 1
       String user = UUID.randomUUID().toString();
       TableWriteImpl<?> write = table.newWrite(user);
       TableCommitImpl commit = table.newCommit(user);
       write.write(GenericRow.of(1, 1));
       commit.commit(1, write.prepareCommit(false, 1));
   
       write.write(GenericRow.of(2, 2));
       write.compact(BinaryRow.EMPTY_ROW, 0, true);
       List<CommitMessage> messages2 = write.prepareCommit(true, 2);
       // the job dies between the append and the compact snapshot of 
identifier 2                                                                    
       commit.commit(2, appendOnly(messages2));
   
       // the writer keeps compacting on top of identifier 2's compaction output
       write.write(GenericRow.of(3, 3));                                        
                                                                     
       write.compact(BinaryRow.EMPTY_ROW, 0, true);
       List<CommitMessage> messages3 = write.prepareCommit(true, 3);
   
       Map<Long, List<CommitMessage>> restored = new HashMap<>();
       restored.put(2L, messages2);
       restored.put(3L, messages3);
       table.newCommit(user).filterAndCommit(restored); // throws
   }
   ```
   
   appendOnly(...) copies each CommitMessageImpl with 
CompactIncrement.emptyIncrement(). 
   
   The test throws:                                            
   
   > IllegalStateException: Trying to delete file data-...-0.parquet for table 
... which is not previously added.
   >   at ConflictDetection.checkDeleteInEntries(ConflictDetection.java:429)
   >   at FileStoreCommitImpl.commit(FileStoreCommitImpl.java:381)
   >   at TableCommitImpl.filterAndCommitMultiple(TableCommitImpl.java:331)
   
   
   ### What doesn't meet your expectations?
   
   Recovery should be idempotent per snapshot, not per identifier. A failure 
between the two snapshots of one identifier should not lose the
   compaction, or make the next identifier's compaction fail.
   
   Job impact: frequent compaction keeps the time window between APPEND and 
COMPACT open almost all the time. A cancellation in that window (for example a 
savepoint timeout) put the job into a restart cascade: 27 restarts, about 35 
minutes, one new table per restart. The changelog produced by a dropped 
compaction (lookup / full-compaction producers) is also lost silently.
   
   ### Anything else?
   
   Proposed fix, in filterCommitted: when committable.identifier() == 
latest.commitIdentifier() and latest.commitKind() != COMPACT, keep a copy of 
the committable that contains only its compact increments, so the compaction is 
recommitted. If any compaction output file no longer exists (for example it was 
removed as an orphan), fall back to today's behaviour and skip it, so recovery 
doesn't get stuck in checkFilesExistence. With the fix, the test above produces 
1:APPEND, 2:APPEND, 2:COMPACT, 3:APPEND, 3:COMPACT and all rows are readable.
   
   The alternative is to write APPEND and COMPACT as a single snapshot. That 
removes the window entirely, but it changes snapshot semantics for streaming 
readers.
   
   ### Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to