JingsongLi commented on code in PR #11181:
URL: https://github.com/apache/paimon/pull/11181#discussion_r4624225756
##########
paimon-core/src/main/java/org/apache/paimon/table/sink/TableCommitImpl.java:
##########
@@ -406,6 +409,20 @@ private void maintain(long identifier, ExecutorService
executor, boolean doExpir
if (batchCommitted) {
maintain(identifier, doExpire);
+ } else if (identifier == COMMIT_IDENTIFIER) {
+ // COMMIT_IDENTIFIER marks the last commit of a bounded input,
such as the end-input
+ // commit of a Flink batch job. close() follows it and would
interrupt or drop async
+ // maintenance, so wait for it. Queue it on the executor instead
of running it here,
+ // because maintenance of earlier commits may still run there on
the same managers.
+ Future<?> future = executor.submit(() -> maintain(identifier,
doExpire));
+ try {
+ future.get();
Review Comment:
[P2] End-input maintenance is skipped on recovery if it fails after the
terminal snapshot is published. I reproduced this on a real append table with
watermark tags and a FileIO that throws once on the tag write.
`filterAndCommitMultiple` throws from the awaited maintenance, but snapshot 1
is durable and the tag is absent. Reopening the committer with the same user
and retrying the exact terminal manifest (as `CommitterOperator` does) returns
0, performs no second tag write and leaves the tag absent. `filterCommitted`
drops the terminal identifier, and the empty retry branch never schedules
maintenance. Please ensure a retry can finish terminal maintenance without
recommitting the data or reporting success while it is still missing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]