sundapeng opened a new pull request, #9424: URL: https://github.com/apache/paimon/pull/9424
### Purpose A Format Table `INSERT OVERWRITE` that replaces the partitions it writes deletes its old data files one request each. On a table holding thousands of files that is thousands of round trips on the driver; #9397 bounds how many are in flight, but not how many there are. This makes that cleanup hand whole batches to `FileIO.deleteFilesInBatch` (#9398). On OSS a 9,384-file overwrite becomes about ten requests instead of 9,384. ### Tests `FormatTableCommitBatchDeleteTest`, 11 cases: batching on the qualifying path, batch bounds, the fall back when the file system deletes one file at a time, a batch failure failing the commit without silently deleting the same files again, and one case per non-qualifying path. `ThreadPoolUtilsTest` 8, `CoreOptionsTest` 16, `FormatTableCommitTest` 43, `FormatTableCommitStatisticsTest` 22, all green on this branch. ### API and Format No public API or format change. `deleteFilesInBatch` is the one added by #9398. ### Documentation None. No new option: the existing `format-table.commit.cleanup-thread-num` gates this too, so setting it to 1 remains a full opt out. --- ### Scope Only a dynamic overwrite of a catalog-managed partitioned table deletes in batches. It replaces exactly the partitions it writes, so it needs no per-file answer about which directories it emptied. A static or whole-table overwrite reports the partitions it cleared and a batch cannot say which files it found, so those keep deleting file by file, and so do truncate, filesystem-discovered tables, unpartitioned tables and `cleanup-thread-num = 1`. The first file is offered on its own. A file system that deletes one file at a time answers without touching storage, so the commit falls back having listed one file rather than a whole batch, and nothing has been deleted yet. After that, batches are bounded, so listing stays as lazy as the file-by-file path: an overwrite that replaces many partitions holds one batch, not every file the table has. Once a batch is handed over the file system either deletes all of it or fails. A failure fails the commit rather than deleting the same files again one by one, which would hide how much of the batch went through. ### Depends on GitHub has no stacked pull requests, so the branch carries its dependencies and Files changed shows them until they merge. This PR's own change is the last commit, `[core] Delete old format table files in batches during dynamic overwrite`. - #9397 `[api][core] Safely parallelize format table overwrite cleanup` (`6d3ef33041`) - #9399 `[core] Parallelize format table file publication` (`01fc733b7d`) - #9398 `[common][oss] Add a strict batch delete capability to FileIO` (`391b197639`, cherry-picked) ### Not measured end to end The request-count reduction is arithmetic, not a benchmark. The production path this came from uses JindoFileIO, which has no batch delete in the deployed runtime, so it stays on the file-by-file path until that is adapted separately. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
