jonathansudhakar1 opened a new pull request, #41482:
URL: https://github.com/apache/beam/pull/41482

   Fixes #41481.
   
   #37003 disabled four tests in `SpannerChangeStreamErrorTest` to unblock the 
Spanner client upgrade in #36995, and #37002 tracked re-enabling them until the 
stale bot closed it in July. This PR re-enables them. The change is test-only; 
no connector code changes.
   
   They failed on stale expectations, not on a connector bug:
   
   - **`testUnavailableExceptionRetries`, `testAbortedExceptionRetries`, 
`testResourceExhaustedRetry`:** `readChangeStream().expand()` reads the 
database dialect while the pipeline is constructed 
([SpannerIO.java#L2404-L2405](https://github.com/apache/beam/blob/d44b8088865011b78a61d419c688c97862ce1914/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java#L2404-L2405)),
 so the mock's sticky error fails that query. The upgraded client surfaces the 
injected status (`UNAVAILABLE`, `ABORTED` or `RESOURCE_EXHAUSTED`) rather than 
`DEADLINE_EXCEEDED`. The tests now expect a `SpannerException` with the 
injected status, the same way the enabled `testUnknownExceptionDoesNotRetry` 
and `testResourceExhaustedRetryWithDefaultSettings` already do.
   - **`testInvalidRecordReceived`:** it failed with `INTERNAL: ... There is no 
result registered for the statement: select option_value from 
information_schema.change_stream_options ...`. It lacked the 
`mockChangeStreamOptions()`, `mockGetPartitionSize(...)` and 
`mockChangePartitionState(...)` setup that its enabled twin 
`testInvalidRecordReceivedWithDefaultSettings` has. It now has the same mocks 
and expectations as the twin (`PipelineExecutionException` with "Field not 
found", and more than zero `ExecuteSqlRequest`s). It still differs from the 
twin in using `getSpannerConfig()`, which sets quick retry settings for 
streaming SQL as well.
   
   I asked about this approach on #37002 on Nov 11 and went ahead with it after 
a week without a reply.
   
   ### Testing
   
   All with JDK 21, on d44b8088865:
   
   - Before the change, with only the `@Ignore`s removed: `./gradlew 
:sdks:java:io:google-cloud-platform:test --tests 
'org.apache.beam.sdk.io.gcp.spanner.changestreams.SpannerChangeStreamErrorTest'`
 gave 7 tests, 4 failed (the messages quoted in #41481).
   - `./gradlew :sdks:java:io:google-cloud-platform:spotlessApply`: no changes.
   - `./gradlew :sdks:java:io:google-cloud-platform:compileJava 
:sdks:java:io:google-cloud-platform:compileTestJava 
:sdks:java:io:google-cloud-platform:checkstyleMain 
:sdks:java:io:google-cloud-platform:checkstyleTest`: BUILD SUCCESSFUL.
   - `./gradlew :sdks:java:io:google-cloud-platform:test --tests 
'org.apache.beam.sdk.io.gcp.spanner.changestreams.SpannerChangeStreamErrorTest'`:
 7 tests, 0 failures, 0 skipped. Then 5 more consecutive runs with `--rerun`: 7 
of 7 passed each time.
   
   ### Open questions
   
   - In the three retry tests (and in the enabled 
`testUnknownExceptionDoesNotRetry` and 
`testResourceExhaustedRetryWithDefaultSettings`), the exception now comes from 
`apply()`, so the code after it (`run()`, waiting for `RUNNING`) is never 
reached. I kept the change minimal and left that code alone. I'm happy to 
remove it here, or to rework these tests in a follow-up so that the sticky 
error hits the read path instead of the dialect query, if you'd prefer that.
   - #21533 (2022, assigned to thiagotnunes) is about an earlier failure of 
`testUnavailableExceptionRetries` that #28751 fixed in 2023. It may be worth 
closing once this lands.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [PR] [Spanner Change Streams] ... jonathansudhakar1 (via GitHub) via github

Reply via email to