sundapeng opened a new pull request, #9399: URL: https://github.com/apache/paimon/pull/9399
### Purpose Builds on https://github.com/apache/paimon/pull/9397, which parallelized the delete half of a format table overwrite commit. This does the publish half. The first commit here belongs to that PR and leaves this diff once it merges. In the profile that motivated both changes, the driver spent roughly 363s deleting old files and then another 242s issuing 9,384 `CompleteUploadPart` requests, one commit message at a time, with no Spark stage running. `format-table.commit.publish-thread-num` (1 to 64, default 64) bounds how many files a commit publishes at once, grouped by target partition: one file at a time within a partition so its input order is preserved, different partitions in parallel. Workers only publish and return their result; partition statistics, staging cleanup and the catalog update stay on the calling thread once every publication has completed. Scope, bounds and failure handling match the cleanup option, and cleanup and publication never overlap. I have not measured the end-to-end effect on that workload yet, so the profile above is what motivated the change rather than a claimed speedup. ### Tests `CoreOptionsTest` for the option, its bounds and its default. `FormatTableCommitPublishTest` for order within a partition and overlap across partitions, the full barrier before statistics, staging cleanup and the catalog update, the first failure draining in-flight publications before abort, caller interrupt, the context classloader of a reused worker, executor rejection, and the single-partition fast path. ### API and Format No public API or file format change. The public `FormatTableCommit` constructor is unchanged and stays serial; the thread count reaches the commit through a package-private constructor that `FormatBatchWriteBuilder` uses. ### Documentation `docs/generated/core_configuration.html` is regenerated for the new option. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
