dragonknightk opened a new pull request, #11181:
URL: https://github.com/apache/paimon/pull/11181

   ### Purpose
   
   Fixes #11180
   
   With `snapshot.expire.execution-mode = async`, `TableCommitImpl` now runs 
maintenance, including snapshot expiration, on the committing thread when the 
commit identifier is `BatchWriteBuilder.COMMIT_IDENTIFIER`. This identifier 
marks the last commit of a bounded input, and `close()` follows that commit. 
Before this change, the maintenance of that commit was queued on the async 
executor, and `close()` calls `shutdownNow()`, which interrupts a running 
expiration or drops one that has not started.
   
   - The Spark compact procedure from #5681 already expires synchronously since 
#6593, because it commits through `commit(List)`, which marks the commit as 
batch. A `COMMIT_IDENTIFIER` commit that does not go through `commit(List)` 
still ran maintenance asynchronously. This covers all Flink bounded commits 
ending with `COMMIT_IDENTIFIER`, and the Flink remove-unexisting-files action.
   - `close()` is unchanged. #5681 proposed `shutdown()` instead of 
`shutdownNow()`, but the maintain thread is a daemon thread, so the queued 
maintenance can still be lost when the process exits after `close()` returns. 
Waiting for it in `close()` would block every caller, including a Flink 
streaming job that is cancelled or stopped while an expiration runs. #5681 also 
proposed skipping expiration in the Spark compact procedure, and recommended an 
internally used IGNORE value of `snapshot.expire.execution-mode` for that. 
#6593 addressed that procedure differently, by expiring synchronously in batch 
commits.
   - The last maintenance runs on the committing thread and may run 
concurrently with an async maintenance still in progress from the previous 
checkpoint. Snapshot expiration already tolerates concurrent runs, and tag 
creation uses `ignoreIfExists`, so this change does not submit that maintenance 
to the executor and wait.
   - Checkpoint commits of streaming jobs keep the async mode. An expiration 
that `close()` interrupts there is picked up by a later expiration.
   - The last commit now behaves as in sync mode: it returns after maintenance, 
and a maintenance failure is thrown to the caller instead of being logged.
   
   ### Tests
   
   - With async mode and the maintain thread kept busy, two checkpoint commits 
keep the first snapshot, so they still expire asynchronously. The following 
end-input commit through `filterAndCommit` with `COMMIT_IDENTIFIER` leaves only 
the latest snapshot after `close()`. `testEndInputCommitExpiresBeforeClose` is 
in `SimpleTableTestBase` and runs in `AppendOnlySimpleTableTest`, 
`PrimaryKeySimpleTableTest`, and `MockRESTSimpleTableTest`. It fails on the 
code before this change.
   
   ## AI Disclosure
   
   Human-led contribution. Grok Bot, working for dragonknightk, assisted with 
this change, including the code, tests, and description. All submitted content 
was reviewed by a human before submission.
   
   (๑•̀ㅂ•́)و✧
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to