shunping opened a new pull request, #40677: URL: https://github.com/apache/beam/pull/40677
## Summary On Dataflow, GcsUtil V2 (java-storage) was up to **36% slower** than V1 (gcsio) for workloads with thousands of small files. This PR makes three changes to `GcsUtilV2` that bring it within noise of V1: 1. **Use the shared HTTP transport.** V2 clients now use the process-wide `Transport.getTransport()`, the same one V1 uses. Before, each client built its own `NetHttpTransport`. With `--gcsPerformanceMetrics`, V2 builds a client on every open and create, so every file got a separate connection pool, and most ranged GETs paid a fresh TLS handshake. 2. **Start upload sessions in the background.** java-storage's `Storage.writer()` blocks while it starts the resumable upload session. `create()` now does that on a background thread, and the channel waits for it at the first `write()` or at `close()`. gcsio does the same for V1. 3. **Raise the default rewrite concurrency from 32 to 128.** java-storage can't batch rewrites, so copy and move run files in parallel. A call never uses more threads than it has files, so only copies and moves of more than 32 files are affected. `--gcsMaxConcurrentRewrites` still overrides the default. ## Performance Setup: ParquetIOLT on Dataflow, `f10_s1k` (40 GB), 3 × e2-standard-2, legacy worker, one run per arm, so treat differences under about ±15% as noise. Each row shows the effect of adding that change on top of the previous ones. V1 served as a control and stayed within ±7%. | Change | Metric | Before → after (V2) | V2 vs V1 gap | |---|---|---|---| | Shared HTTP transport | Read, 4,096 shards | 1,372 → 1,187 s | **+14% → +0.3%** | | | Glob, 4 pages | 2,255 → 2,216 ms | +365 → +185 ms | | Background upload sessions | Time per create, 4,096 shards | 85 → 2.2 ms (summed 348 → 9 s) | V1: 1.3 ms | | | Write, 4,096 shards | 455 → 406 s | **+20% → +9%** | | Rewrite concurrency 128 | Rename, 4,096 shards (summed) | 36.3 → 13.0 s | 3.9× → 1.5× V1 | | | Write, 4,096 shards | 406 → 371 s | **+9% → +4.5%** | All three together, compared with master: | Shards | Write gap (V2 vs V1) | Read gap (V2 vs V1) | |---|---|---| | 4,096 | +15% → **+4.5%** | +14% → **parity** | | 16,384 | +36% → **+3.9%** (create 1,537 → 14 s summed; rename 77.7 → 33.6 s) | not yet measured | Bytes and GCS request counts are unchanged; only connection setup, session-start latency and parallelism changed. ## Behavior changes - `GcsUtilV2.create()` no longer reports errors from starting the upload session. They surface at the first `write()` or at `close()`, as an `IOException` (e.g. `AccessDeniedException`), the same as V1. - The `GcsInsert` API-request metric is recorded when the session start actually succeeds or fails, not when `create()` returns. - The default for `--gcsMaxConcurrentRewrites` is now 128 (it was 32), but the actual number of threads is `MIN(total_number_of_files, maxCurrrentRewrites)`. ## Notes - The transport change matters most with `--gcsPerformanceMetrics`, where V2 builds a client for every open and create. Without metrics V2 already uses one shared client, so the transport change mostly helps glob. - Rename is still about 1.5× V1 per file, because V1 batches its rewrites. Batching V2's destination lookups and source deletes with `StorageBatch` is a possible follow-up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
