jonathansudhakar1 opened a new pull request, #41482: URL: https://github.com/apache/beam/pull/41482
Fixes #41481. #37003 disabled four tests in `SpannerChangeStreamErrorTest` to unblock the Spanner client upgrade in #36995, and #37002 tracked re-enabling them until the stale bot closed it in July. This PR re-enables them. The change is test-only; no connector code changes. They failed on stale expectations, not on a connector bug: - **`testUnavailableExceptionRetries`, `testAbortedExceptionRetries`, `testResourceExhaustedRetry`:** `readChangeStream().expand()` reads the database dialect while the pipeline is constructed ([SpannerIO.java#L2404-L2405](https://github.com/apache/beam/blob/d44b8088865011b78a61d419c688c97862ce1914/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java#L2404-L2405)), so the mock's sticky error fails that query. The upgraded client surfaces the injected status (`UNAVAILABLE`, `ABORTED` or `RESOURCE_EXHAUSTED`) rather than `DEADLINE_EXCEEDED`. The tests now expect a `SpannerException` with the injected status, the same way the enabled `testUnknownExceptionDoesNotRetry` and `testResourceExhaustedRetryWithDefaultSettings` already do. - **`testInvalidRecordReceived`:** it failed with `INTERNAL: ... There is no result registered for the statement: select option_value from information_schema.change_stream_options ...`. It lacked the `mockChangeStreamOptions()`, `mockGetPartitionSize(...)` and `mockChangePartitionState(...)` setup that its enabled twin `testInvalidRecordReceivedWithDefaultSettings` has. It now has the same mocks and expectations as the twin (`PipelineExecutionException` with "Field not found", and more than zero `ExecuteSqlRequest`s). It still differs from the twin in using `getSpannerConfig()`, which sets quick retry settings for streaming SQL as well. I asked about this approach on #37002 on Nov 11 and went ahead with it after a week without a reply. ### Testing All with JDK 21, on d44b8088865: - Before the change, with only the `@Ignore`s removed: `./gradlew :sdks:java:io:google-cloud-platform:test --tests 'org.apache.beam.sdk.io.gcp.spanner.changestreams.SpannerChangeStreamErrorTest'` gave 7 tests, 4 failed (the messages quoted in #41481). - `./gradlew :sdks:java:io:google-cloud-platform:spotlessApply`: no changes. - `./gradlew :sdks:java:io:google-cloud-platform:compileJava :sdks:java:io:google-cloud-platform:compileTestJava :sdks:java:io:google-cloud-platform:checkstyleMain :sdks:java:io:google-cloud-platform:checkstyleTest`: BUILD SUCCESSFUL. - `./gradlew :sdks:java:io:google-cloud-platform:test --tests 'org.apache.beam.sdk.io.gcp.spanner.changestreams.SpannerChangeStreamErrorTest'`: 7 tests, 0 failures, 0 skipped. Then 5 more consecutive runs with `--rerun`: 7 of 7 passed each time. ### Open questions - In the three retry tests (and in the enabled `testUnknownExceptionDoesNotRetry` and `testResourceExhaustedRetryWithDefaultSettings`), the exception now comes from `apply()`, so the code after it (`run()`, waiting for `RUNNING`) is never reached. I kept the change minimal and left that code alone. I'm happy to remove it here, or to rework these tests in a follow-up so that the sticky error hits the read path instead of the dialect query, if you'd prefer that. - #21533 (2022, assigned to thiagotnunes) is about an earlier failure of `testUnavailableExceptionRetries` that #28751 fixed in 2023. It may be worth closing once this lands. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
