hbgstc123 opened a new pull request, #8927: URL: https://github.com/apache/paimon/pull/8927
### Purpose **Problem** When Spark speculative execution is enabled, Paimon Spark writers use `useCommitCoordinator=false`. Multiple task attempts can all finish writing and call `prepareCommit()`, but Spark only commits the winner's `WriterCommitMessage`. Loser attempts leave data/index files on disk that are never referenced by any snapshot — orphan files. **Why it happens** Spark kills the loser attempt by interrupting the task thread. Paimon writers do not abort prepared files on interruption today, and core abort/delete paths can fail with `InterruptedIOException` while the thread is interrupted, so loser files are often left behind until orphan cleanup runs (if ever). **What this PR does** - Add task-side `SparkAttemptCleanup` to detect interruption and abort unprepared/prepared files before the attempt returns successfully. - Wire cleanup into Spark V1/V2 writers, data-evolution writes, format-table writes, and compact procedure. - Make core abort paths interrupt-tolerant (`FileIO.deleteQuietlyIgnoringInterrupt`, `FileStoreCommitImpl.abort` index cleanup, writer abort executors). - Register prepared files incrementally during `prepareCommit` so a mid-attempt kill can still abort already-finished files. ### Tests - `SparkAttemptCleanupTest`, `DataEvolutionTableDataWriteTest` - `FileStoreCommitAbortTest`, `FileIOTest`, `DeletionVectorsIndexFileTest` - `SparkSpeculativeWriteITCase` (DSv2/V1 insert with speculation; loser cleanup and orphan-clean fallback) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
