mxtymoshyk commented on code in PR #39938:
URL: https://github.com/apache/beam/pull/39938#discussion_r4379892453
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java:
##########
@@ -4227,10 +4236,15 @@ private <DestinationT> WriteResult continueExpandTyped(
batchLoads.setMaxFilesPerPartition(getMaxFilesPerPartition());
batchLoads.setMaxBytesPerPartition(getMaxBytesPerPartition());
- // When running in streaming (unbounded mode) we want to retry failed
load jobs
- // indefinitely. Failing the bundle is expensive, so we set a fairly
high limit on retries.
- if (IsBounded.UNBOUNDED.equals(input.isBounded())) {
- batchLoads.setMaxRetryJobs(getMaxRetryJobs());
+ // an explicit withMaxRetryJobs applies to batch and streaming alike.
left unset, streaming
+ // retries a failed load job far more often than batch does: failing
the bundle in streaming
+ // is expensive, so we would rather keep retrying the job than hand
the work back to the
+ // runner. batch leaves BatchLoads on its own lower default
+ Integer maxRetryJobs = getMaxRetryJobs();
Review Comment:
Small update: I moved the cutoff from 2.77.0 to 2.78.0 (1a3f527), since this
change ships in 2.78.0. The logic is the same. A row from before 2.78.0 holding
exactly 1000 is treated as unset, and rows from 2.78.0 onward keep an explicit
`withMaxRetryJobs(1000)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
