dragonknightk opened a new pull request, #11181: URL: https://github.com/apache/paimon/pull/11181
### Purpose Fixes #11180 With `snapshot.expire.execution-mode = async`, `TableCommitImpl` now runs maintenance, including snapshot expiration, on the committing thread when the commit identifier is `BatchWriteBuilder.COMMIT_IDENTIFIER`. This identifier marks the last commit of a bounded input, and `close()` follows that commit. Before this change, the maintenance of that commit was queued on the async executor, and `close()` calls `shutdownNow()`, which interrupts a running expiration or drops one that has not started. - The Spark compact procedure from #5681 already expires synchronously since #6593, because it commits through `commit(List)`, which marks the commit as batch. A `COMMIT_IDENTIFIER` commit that does not go through `commit(List)` still ran maintenance asynchronously. This covers all Flink bounded commits ending with `COMMIT_IDENTIFIER`, and the Flink remove-unexisting-files action. - `close()` is unchanged. #5681 proposed `shutdown()` instead of `shutdownNow()`, but the maintain thread is a daemon thread, so the queued maintenance can still be lost when the process exits after `close()` returns. Waiting for it in `close()` would block every caller, including a Flink streaming job that is cancelled or stopped while an expiration runs. #5681 also proposed skipping expiration in the Spark compact procedure, and recommended an internally used IGNORE value of `snapshot.expire.execution-mode` for that. #6593 addressed that procedure differently, by expiring synchronously in batch commits. - The last maintenance runs on the committing thread and may run concurrently with an async maintenance still in progress from the previous checkpoint. Snapshot expiration already tolerates concurrent runs, and tag creation uses `ignoreIfExists`, so this change does not submit that maintenance to the executor and wait. - Checkpoint commits of streaming jobs keep the async mode. An expiration that `close()` interrupts there is picked up by a later expiration. - The last commit now behaves as in sync mode: it returns after maintenance, and a maintenance failure is thrown to the caller instead of being logged. ### Tests - With async mode and the maintain thread kept busy, two checkpoint commits keep the first snapshot, so they still expire asynchronously. The following end-input commit through `filterAndCommit` with `COMMIT_IDENTIFIER` leaves only the latest snapshot after `close()`. `testEndInputCommitExpiresBeforeClose` is in `SimpleTableTestBase` and runs in `AppendOnlySimpleTableTest`, `PrimaryKeySimpleTableTest`, and `MockRESTSimpleTableTest`. It fails on the code before this change. ## AI Disclosure Human-led contribution. Grok Bot, working for dragonknightk, assisted with this change, including the code, tests, and description. All submitted content was reviewed by a human before submission. (๑•̀ㅂ•́)و✧ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
