damccorm opened a new pull request, #41120: URL: https://github.com/apache/beam/pull/41120
### Summary In multiple recent runs (, , , ), GCS tests using `GcsUtilV2` failed on transient HTTP 503 `StorageException` errors (`We encountered an internal error. Please try again.` or `server(s) are not responding`) during `GcsUtilV2.rewriteOne`, `GcsUtilV2.remove`, and `GcsUtilV2.removeBucket`: - https://github.com/apache/beam/actions/runs/42586715354 (`GcsUtilIT.testGetFileSizes` / `removeBucket`) - https://github.com/apache/beam/actions/runs/42596116347 (`WordCountIT` / `GcsUtilV2.rewriteOne` -> `remove`) - https://github.com/apache/beam/actions/runs/42594496738 (`TFRecordIOIT.testWriteAndReadAll` / `GcsUtilV2.remove`) - https://github.com/apache/beam/actions/runs/42576842339 (`WindowedWordCountIT` / `GcsUtilV2.remove`) There are two reasons why `GcsUtilV2` did not retry these transient 503 errors: 1. `StorageOptions.newBuilder()` defaults to `StorageRetryStrategy.getDefaultStorageRetryStrategy()`, which treats unconditional `storage().delete(BlobId)` and `storage().copy(CopyRequest)` (without generation preconditions) as non-idempotent and does not retry transient 5xx/429 errors. Setting `StorageRetryStrategy.getUniformStorageRetryStrategy()` matches `GcsUtilV1`'s `RetryHttpRequestInitializer` behavior. 2. `StorageBatch.submit()` in `getBlobs()` and `remove()` sends a single multipart HTTP request and does not retry individual sub-requests when a sub-response inside the batch fails with a retryable `StorageException` (`e.isRetryable()`). Falling back to an individual `storage().get(...)` / `storage().delete(...)` on a retryable batch sub-request exception allows the client's retry strategy to retry the failed item. ### Verification - Triggered `PreCommit Java` on branch `fix/gcsutil-v2-retry-hardening`: https://github.com/apache/beam/actions/runs/42655311292 - Triggered `PostCommit Java Examples Direct` on branch `fix/gcsutil-v2-retry-hardening`: https://github.com/apache/beam/actions/runs/42655317142 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
