sundapeng opened a new pull request, #9397: URL: https://github.com/apache/paimon/pull/9397
### Purpose `INSERT OVERWRITE` on a catalog-managed Format Table finishes its Spark stages and then sits in the driver. `FormatTableCommit` deletes the old data files of the target partitions one at a time, so a commit that replaces N files pays N sequential round trips to object storage before it can publish anything. A profile of one production statement: it ran 857.9s, the last Spark task finished 605.8s before it returned, and no Spark stage ran in that window. In the same window object storage recorded one driver client issuing 12,700 metadata, 9,384 PUT and 9,384 DELETE requests, the deletes spanning roughly 363s of it. Reading these tables is already parallel - `format-table.scan.list-parallelism` (default 64) lists partition files concurrently during split planning - and this gives the delete side the same treatment. `format-table.commit.cleanup-thread-num` (1 to 64, default 64) bounds how many old data files a commit deletes at once: one task per file and a sliding window of at most 64 in flight per commit, on one FIFO executor shared by concurrent commits so a large commit cannot lock a small one out. The first failure stops handing out work, every accepted task is drained before the commit fails, and the primary exception is picked by input order with the rest attached as suppressed. Cleanup still completes before anything is published. The scope matches the read side: only an internal Format Table whose partitions the catalog manages, that is `partitionManager != null && !partitionKeys.isEmpty()`. Filesystem-discovered format tables, unpartitioned format tables, ordinary Paimon tables and the existing public `FormatTableCommit` constructor keep the serial path, and `TRUNCATE TABLE` and `TRUNCATE PARTITION` stay serial as well. I have not measured the end-to-end effect on that workload yet, so the profile above is what motivated the change rather than a claimed speedup. ### Tests `CoreOptionsTest` for the option, its bounds and its default. `FormatTableCommitTest` for the concurrency a catalog-managed builder actually reaches and the 64 ceiling, the serial path for explicit `1`, filesystem-discovered, unpartitioned and old-constructor tables, the cleanup to publish barrier, the first failure stopping submission and draining accepted work, the primary exception chosen by input position with the others suppressed, interrupt flag restoration, an abort failure not masking the original one, fairness between a large and a small commit on the shared executor, one concurrency window across partition roots, and no partition root listed before the first deletes complete. ### API and Format No public API or file format change. The public `FormatTableCommit` constructor is unchanged and stays serial; the thread count reaches the commit through a package-private constructor that `FormatBatchWriteBuilder` uses. ### Documentation `docs/generated/core_configuration.html` is regenerated for the new option. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
