JingsongLi commented on code in PR #11181:
URL: https://github.com/apache/paimon/pull/11181#discussion_r4583223022
##########
paimon-core/src/main/java/org/apache/paimon/table/sink/TableCommitImpl.java:
##########
@@ -404,7 +404,9 @@ private void maintain(long identifier, ExecutorService
executor, boolean doExpir
throw new RuntimeException(maintainError.get());
}
- if (batchCommitted) {
+ // COMMIT_IDENTIFIER marks the last commit of a bounded input, such as
the end-input commit
+ // of a Flink batch job. close() follows it and would interrupt or
drop async maintenance.
+ if (batchCommitted || identifier == COMMIT_IDENTIFIER) {
maintain(identifier, doExpire);
Review Comment:
[P1] Serialize final maintenance with already-running asynchronous
maintenance
This direct call can enter the same maintenance managers as the previous
checkpoint's executor task. With decoupled changelog retention, the shared
`ChangelogDeletion` tag skipper closes over `cachedTagDataFiles`: a second pass
can replace the retained-tag-1 cache with tag 3 while the first pass is between
planning and filtering its deletions. The first pass then deletes a file still
required by tag 1.
I reproduced this with six real primary-key snapshots (overwrites at 2 and
4), retained tags at 1 and 3, snapshot retention min/max 1 and changelog
retention max 2. Holding the old pass after planning changelog 2, running the
final `COMMIT_IDENTIFIER` pass, then resuming the old pass removes the
protected data file. The tag remains registered; reloading its persisted
metadata and reading it throws `FileNotFoundException`. Both cases pass on
clean base `9b6c997198`; the same-head serialized checkpoint control also
retains and reads the file.
There is a second manifestation with watermark automatic tags: three real
append commits persist tags for snapshots `[1,3]` under overlap, versus
`[1,2,3]` in the serialized control, because both runs mutate the same
`TagAutoCreation.nextSnapshot`. `ignoreIfExists` protects duplicate tag
creation, not this progress state.
Please queue the final maintenance behind preceding maintenance and wait for
completion, or otherwise serialize access to all of these shared managers. It
still needs to finish before close, while preserving the existing ordering of
maintenance passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]