JingsongLi commented on code in PR #11181:
URL: https://github.com/apache/paimon/pull/11181#discussion_r4624225756


##########
paimon-core/src/main/java/org/apache/paimon/table/sink/TableCommitImpl.java:
##########
@@ -406,6 +409,20 @@ private void maintain(long identifier, ExecutorService 
executor, boolean doExpir
 
         if (batchCommitted) {
             maintain(identifier, doExpire);
+        } else if (identifier == COMMIT_IDENTIFIER) {
+            // COMMIT_IDENTIFIER marks the last commit of a bounded input, 
such as the end-input
+            // commit of a Flink batch job. close() follows it and would 
interrupt or drop async
+            // maintenance, so wait for it. Queue it on the executor instead 
of running it here,
+            // because maintenance of earlier commits may still run there on 
the same managers.
+            Future<?> future = executor.submit(() -> maintain(identifier, 
doExpire));
+            try {
+                future.get();

Review Comment:
   [P2] End-input maintenance is skipped on recovery if it fails after the 
terminal snapshot is published. I reproduced this on a real append table with 
watermark tags and a FileIO that throws once on the tag write. 
`filterAndCommitMultiple` throws from the awaited maintenance, but snapshot 1 
is durable and the tag is absent. Reopening the committer with the same user 
and retrying the exact terminal manifest (as `CommitterOperator` does) returns 
0, performs no second tag write and leaves the tag absent. `filterCommitted` 
drops the terminal identifier, and the empty retry branch never schedules 
maintenance. Please ensure a retry can finish terminal maintenance without 
recommitting the data or reporting success while it is still missing.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to