wangzhigang1999 opened a new issue, #11111: URL: https://github.com/apache/paimon/issues/11111
### Search before asking - [x] I searched the [issues](https://github.com/apache/paimon/issues) for abort cleanup and found no report for this path. ### Motivation A failed write to a regular Paimon table can spend a long time cleaning up files before the write error returns to the caller. [`FileStoreCommitImpl.abort`](https://github.com/apache/paimon/blob/a9781112a11a9a7d7d4147644add9af16f2359bf/paimon-core/src/main/java/org/apache/paimon/operation/FileStoreCommitImpl.java#L732-L769) calls `FileIO.deleteQuietly` for each file on the caller thread, so delete latency accumulates with the number of files. I reproduced this on commit `a9781112a11a9a7d7d4147644add9af16f2359bf` with an ordinary bucket-unaware append table. The Java batch writer wrote one Parquet file per partition and returned real commit messages from `prepareCommit()`. A local FileIO wrapper added 20 ms of latency to each delete. Timing only the call to `BatchTableCommit.abort(messages)` gave these medians over three runs. | Files to delete | Median abort time | | ---: | ---: | | 16 | 0.436 s | | 64 | 1.715 s | | 128 | 3.362 s | Every file was deleted through a separate call, and peak in-flight deletes remained 1 with `file-operation.thread-num=8`. A previously committed file and its row remained readable. The latency was injected locally. ### Solution Consider using bounded concurrency for the individual delete calls in `FileStoreCommitImpl.abort`. The caller would still wait for cleanup to finish, and cleanup failures would retain the current best-effort behavior. Keep the existing rules for which files belong to the failed write, including the protection for files reused by compaction in #10582. There is prior work on parallel file operations for snapshot expiration (#8241) and Format Table overwrite (#9397). ### Anything else? The failure path can be triggered with Spark 3.5.9 and the Paimon Spark 3.5 jar built from the commit above. Run Spark with `local[1]`, `spark.driver.maxResultSize=12k`, `spark.sql.adaptive.enabled=false`, and `spark.paimon.write.use-v2-write=true`. Enable `org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions` and register a Paimon catalog named `paimon` using a temporary local warehouse. ```sql CREATE DATABASE IF NOT EXISTS paimon.probe; CREATE TABLE paimon.probe.t (id INT, p INT) USING paimon PARTITIONED BY (p) TBLPROPERTIES ('bucket' = '-1', 'write-only' = 'true', 'file.format' = 'parquet'); INSERT INTO paimon.probe.t VALUES (-1, -1); INSERT INTO paimon.probe.t SELECT CAST(id AS INT), CAST(CASE WHEN id < 24 THEN id % 12 ELSE 100 + id % 14 END AS INT) FROM range(0, 48, 1, 2); ``` The two task results were each below the configured limit, but Spark reported 15.0 KiB in total and invoked Paimon's abort. The 12 files whose commit messages reached Paimon were deleted sequentially. The original snapshot and row remained readable. The 14 files from the task whose result was rejected remained on disk, outside the set of files described by the received commit messages. A separate local probe wired the same Paimon abort entry point through Jindo 6.9.1 and 6.10.9 to a mock OSS endpoint. Both runs issued one HTTP DELETE per key. ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
