JingsongLi commented on code in PR #11181:
URL: https://github.com/apache/paimon/pull/11181#discussion_r4583223022


##########
paimon-core/src/main/java/org/apache/paimon/table/sink/TableCommitImpl.java:
##########
@@ -404,7 +404,9 @@ private void maintain(long identifier, ExecutorService 
executor, boolean doExpir
             throw new RuntimeException(maintainError.get());
         }
 
-        if (batchCommitted) {
+        // COMMIT_IDENTIFIER marks the last commit of a bounded input, such as 
the end-input commit
+        // of a Flink batch job. close() follows it and would interrupt or 
drop async maintenance.
+        if (batchCommitted || identifier == COMMIT_IDENTIFIER) {
             maintain(identifier, doExpire);

Review Comment:
   [P1] Serialize final maintenance with already-running asynchronous 
maintenance
   
   This direct call can enter the same maintenance managers as the previous 
checkpoint's executor task. With decoupled changelog retention, the shared 
`ChangelogDeletion` tag skipper closes over `cachedTagDataFiles`: a second pass 
can replace the retained-tag-1 cache with tag 3 while the first pass is between 
planning and filtering its deletions. The first pass then deletes a file still 
required by tag 1.
   
   I reproduced this with six real primary-key snapshots (overwrites at 2 and 
4), retained tags at 1 and 3, snapshot retention min/max 1 and changelog 
retention max 2. Holding the old pass after planning changelog 2, running the 
final `COMMIT_IDENTIFIER` pass, then resuming the old pass removes the 
protected data file. The tag remains registered; reloading its persisted 
metadata and reading it throws `FileNotFoundException`. Both cases pass on 
clean base `9b6c997198`; the same-head serialized checkpoint control also 
retains and reads the file.
   
   There is a second manifestation with watermark automatic tags: three real 
append commits persist tags for snapshots `[1,3]` under overlap, versus 
`[1,2,3]` in the serialized control, because both runs mutate the same 
`TagAutoCreation.nextSnapshot`. `ignoreIfExists` protects duplicate tag 
creation, not this progress state.
   
   Please queue the final maintenance behind preceding maintenance and wait for 
completion, or otherwise serialize access to all of these shared managers. It 
still needs to finish before close, while preserving the existing ordering of 
maintenance passes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to