bvolpato opened a new pull request, #40937: URL: https://github.com/apache/beam/pull/40937
`SpannerIO.readChangeStream()` adds `ABORTED` to its retryable codes for its streaming change stream queries. `MetadataSpannerConfigFactory` copied those codes to the metadata connection, where they also apply to `Commit`. When Spanner aborted a metadata transaction, the client resent the commit with the same aborted transaction ID until retries ran out, instead of returning the error so that `TransactionRunner` could run the transaction again. With the real Spanner client and a mock server, one aborted transaction took about 50 seconds to recover and sent about 20 commit RPCs. Sometimes the last attempt starts with only milliseconds of the 60 second commit deadline left. It then fails with `DEADLINE_EXCEEDED`, which `TransactionRunner` does not replay, so the pipeline fails. This matches the `SpannerChangeStreamPlacementTableIT` failures in the GCP IO precommit ([example](https://github.com/apache/beam/actions/runs/40074964342/job/120816244820), [example](https://github.com/apache/beam/actions/runs/41021022224/job/123948121010)). They report `CallOptions deadline exceeded after 0.009149773s` with a request ID on attempt 20 or 21, and the failed tests take about 66 to 69 seconds. The CI logs do not include the status of the earlier attempts, so `ABORTED` as the trigger is inferred rather than observed. Repeated `UNAVAILABLE` could use up the same deadline, and this change does not address that case. This change removes `ABORTED` from the metadata config. The primary config keeps it for change stream queries, and `UNAVAILABLE` is still retried on both. With 250 ms of simulated commit latency on the mock server, the old code reproduced the CI failure, with about 8 ms left on the last attempt. With this change, an aborted transaction recovers in about 0.5 seconds with one replay. The new regression test fails before the fix and passes after it. The Spanner unit tests, Spotless, and Checkstyle ran locally. The change stream integration tests need GCP, so they were not run locally. The GCP IO precommit on this PR runs them. ```sh ./gradlew :sdks:java:io:google-cloud-platform:test --tests '*MetadataSpannerConfigFactoryTest*' ``` <!-- Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). - [x] Replace these instructions with a description of your change and why you have made it. - [ ] Mention the appropriate issue in your description (for example: `addresses #123` or `fixes #456`). - [x] Update `CHANGES.md` with noteworthy changes. See [CONTRIBUTING.md]() for more details. --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
