wangzhigang1999 opened a new issue, #11111:
URL: https://github.com/apache/paimon/issues/11111

   ### Search before asking
   
   - [x] I searched the [issues](https://github.com/apache/paimon/issues) for 
abort cleanup and found no report for this path.
   
   ### Motivation
   
   A failed write to a regular Paimon table can spend a long time cleaning up 
files before the write error returns to the caller. 
[`FileStoreCommitImpl.abort`](https://github.com/apache/paimon/blob/a9781112a11a9a7d7d4147644add9af16f2359bf/paimon-core/src/main/java/org/apache/paimon/operation/FileStoreCommitImpl.java#L732-L769)
 calls `FileIO.deleteQuietly` for each file on the caller thread, so delete 
latency accumulates with the number of files.
   
   I reproduced this on commit `a9781112a11a9a7d7d4147644add9af16f2359bf` with 
an ordinary bucket-unaware append table. The Java batch writer wrote one 
Parquet file per partition and returned real commit messages from 
`prepareCommit()`. A local FileIO wrapper added 20 ms of latency to each 
delete. Timing only the call to `BatchTableCommit.abort(messages)` gave these 
medians over three runs.
   
   | Files to delete | Median abort time |
   | ---: | ---: |
   | 16 | 0.436 s |
   | 64 | 1.715 s |
   | 128 | 3.362 s |
   
   Every file was deleted through a separate call, and peak in-flight deletes 
remained 1 with `file-operation.thread-num=8`. A previously committed file and 
its row remained readable. The latency was injected locally.
   
   ### Solution
   
   Consider using bounded concurrency for the individual delete calls in 
`FileStoreCommitImpl.abort`. The caller would still wait for cleanup to finish, 
and cleanup failures would retain the current best-effort behavior. Keep the 
existing rules for which files belong to the failed write, including the 
protection for files reused by compaction in #10582.
   
   There is prior work on parallel file operations for snapshot expiration 
(#8241) and Format Table overwrite (#9397).
   
   ### Anything else?
   
   The failure path can be triggered with Spark 3.5.9 and the Paimon Spark 3.5 
jar built from the commit above. Run Spark with `local[1]`, 
`spark.driver.maxResultSize=12k`, `spark.sql.adaptive.enabled=false`, and 
`spark.paimon.write.use-v2-write=true`. Enable 
`org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions` and register 
a Paimon catalog named `paimon` using a temporary local warehouse.
   
   ```sql
   CREATE DATABASE IF NOT EXISTS paimon.probe;
   CREATE TABLE paimon.probe.t (id INT, p INT) USING paimon PARTITIONED BY (p)
   TBLPROPERTIES ('bucket' = '-1', 'write-only' = 'true', 'file.format' = 
'parquet');
   INSERT INTO paimon.probe.t VALUES (-1, -1);
   
   INSERT INTO paimon.probe.t
   SELECT CAST(id AS INT),
          CAST(CASE WHEN id < 24 THEN id % 12 ELSE 100 + id % 14 END AS INT)
   FROM range(0, 48, 1, 2);
   ```
   
   The two task results were each below the configured limit, but Spark 
reported 15.0 KiB in total and invoked Paimon's abort. The 12 files whose 
commit messages reached Paimon were deleted sequentially. The original snapshot 
and row remained readable. The 14 files from the task whose result was rejected 
remained on disk, outside the set of files described by the received commit 
messages.
   
   A separate local probe wired the same Paimon abort entry point through Jindo 
6.9.1 and 6.10.9 to a mock OSS endpoint. Both runs issued one HTTP DELETE per 
key.
   
   ### Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to