mxtymoshyk opened a new pull request, #40793: URL: https://github.com/apache/beam/pull/40793
Fixes #40792. Many BigQuery integration tests check their output with `BigqueryClient.queryUnflattened`. It polled `jobs.getQueryResults` with no retry, so a single transient 503 `backendError` failed a test after its pipeline had already succeeded. This happened in `StorageApiDataTriggeredSchemaUpdateIT` on #40790. - `queryUnflattened` now polls through a new `getQueryResultsWithRetries`, which uses the same backoff as the other `BigqueryClient` methods. It retries 5xx and 429 errors and throws other 4xx errors, such as a failed query, right away. - The temporary dataset is deleted in a `finally` block. Before, the method skipped the delete when the query returned no rows or threw. `BigqueryClientTest` has 3 new tests: a retry after a 503, no retry on a 400, and cleanup when the query returns no rows. All 3 fail without this change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
