Stephen0421 opened a new pull request, #9345: URL: https://github.com/apache/paimon/pull/9345
### Purpose The files written by a compaction whose result is never committed are left behind as orphan files. There are two ways to lose track of them: 1. `CompactFutureManager#cancelCompaction` cancels the future with `taskFuture.cancel(true)`. When the cancellation wins the race against the completion of the task, `FutureTask` silently drops the value returned by `call()`. The files of that result are already on disk, but the reference to them is gone, so no caller can account for them. 2. A task failing halfway lost the output of the steps which had already finished, because only the returned result was ever visible to the caller. This change makes the ownership of the produced files explicit: * `CompactTask` accumulates its output into a single result, so a partial result of a failed task is still known. * Publishing a finished result and declaring it cancelled are mutually exclusive. Either the task publishes first and the manager can still salvage the result after `FutureTask` dropped it, or the cancellation wins and the task is the one which deletes the files. There is no state in which both sides believe the other one takes care of them. * A cancelled or failed task deletes its own output. Output files which are the very same physical file as an input (a file only upgraded to another level) are kept, since previous snapshots still require them. * The cleanup clears the cancellation interrupt while deleting and restores it afterwards. File systems backed by RPC fail their calls immediately while the interrupt flag is set, and `deleteQuietly` only warns, so the cleanup would silently leave exactly the orphan files it is supposed to remove. The deletion hooks (`CompactRewriter#deleteProduced` and the `delete` of the append rewriters) are abstract rather than defaulted to a no-op, so a rewriter which produces files and forgets to implement them is a compile error instead of a runtime leak. Note that only file-side cleanup is performed. In-memory side effects applied during compaction (deletion-vector removals, clustering key-index updates) are not rolled back; the invariant that a cancelled result must not be consumed by the same writer's `prepareCommit` is documented on `CompactTask#cancel`. Today `cancelCompaction` is only reachable from writer `close()`, where the maintainer is thrown away together with the writer. ### Tests New `CompactCancellationTest` covers the discarded-output paths: * a cancelled task and a task failing in a later step delete their output; * a file which is only upgraded is not deleted; * the deletion file of a discarded result is cleaned up; * a finished result is not lost when the cancellation wins the `FutureTask` race; * a cancellation landing exactly at the point where the task is about to publish its result, after the caller has already given up on it; * the cleanup runs with the interrupt flag cleared and restores it afterwards; * a 500 iteration loop racing a real cancellation against the completion of the task, asserting the invariant that the output is either reported to the caller or deleted by the task. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
