mxtymoshyk commented on code in PR #39938:
URL: https://github.com/apache/beam/pull/39938#discussion_r4379892453
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java:
##########
@@ -4227,10 +4236,15 @@ private <DestinationT> WriteResult continueExpandTyped(
batchLoads.setMaxFilesPerPartition(getMaxFilesPerPartition());
batchLoads.setMaxBytesPerPartition(getMaxBytesPerPartition());
- // When running in streaming (unbounded mode) we want to retry failed
load jobs
- // indefinitely. Failing the bundle is expensive, so we set a fairly
high limit on retries.
- if (IsBounded.UNBOUNDED.equals(input.isBounded())) {
- batchLoads.setMaxRetryJobs(getMaxRetryJobs());
+ // an explicit withMaxRetryJobs applies to batch and streaming alike.
left unset, streaming
+ // retries a failed load job far more often than batch does: failing
the bundle in streaming
+ // is expensive, so we would rather keep retrying the job than hand
the work back to the
+ // runner. batch leaves BatchLoads on its own lower default
+ Integer maxRetryJobs = getMaxRetryJobs();
Review Comment:
Moved the cutoff from 2.77.0 to 2.78.0 in 1a3f527, since this ships in
2.78.0. Nothing else changed: `fromConfigRow` ignores a 1000 from a pre-2.78.0
row and keeps an explicit `withMaxRetryJobs(1000)` from 2.78.0 rows onward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
