shunping opened a new pull request, #40376: URL: https://github.com/apache/beam/pull/40376
`GcsUtilV2.rewriteHelper` previously processed files sequentially, making up to three dependent RPCs per file (target lookup, rewrite, and source delete). Unlike V1, which batches rewrites and runs batches concurrently, V2's `java-storage` client does not support rewrite in `StorageBatch`, causing `rename` to be ~6x slower than V1 in 64-shard `ParquetIOLT` benchmarks. - **`GcsUtilV2`**: extract per-file logic into `rewriteOne` and execute multi-file `copy`/`move` calls concurrently on a bounded thread pool (up to 32 threads). Propagate the caller's `MetricsContainer` to worker threads, cancel unstarted tasks on the first failure, and rethrow the original exception type (`FileNotFoundException`, `FileAlreadyExistsException`, `AccessDeniedException`, etc.) with subsequent failures attached as suppressed. - **`GcsUtilV2Test`**: add `testV2RenameRewritesFilesConcurrently` and `testV2RenameFailuresAreReportedPerFile`. In the 64-shard `ParquetIOLT` (`f1_s1k`) write benchmark on Dataflow, V2 `gcs_op_rename_msec` dropped from ~20.6s to ~1.0s (vs ~2.3s–2.9s on V1). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
