Abacn commented on code in PR #33540:
URL: https://github.com/apache/beam/pull/33540#discussion_r1923971579
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageStreamSource.java:
##########
@@ -368,8 +368,13 @@ public synchronized BigQueryStorageStreamSource<T>
getCurrentSource() {
.setName(source.readStream.getName())
.setFraction((float) fraction)
.build();
-
- SplitReadStreamResponse splitResponse =
storageClient.splitReadStream(splitRequest);
+ SplitReadStreamResponse splitResponse;
+ try {
+ splitResponse = storageClient.splitReadStream(splitRequest);
+ } catch (Exception e) {
Review Comment:
I have thought about this question. It makes sense to make this generic but
is also a much larger scope of changes.
For reference, Code pointer:
https://github.com/apache/beam/blob/58236b2f115d28993cd3ee50c6a319fe21761ba1/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowWorkProgressUpdater.java#L160)
https://github.com/apache/beam/blob/58236b2f115d28993cd3ee50c6a319fe21761ba1/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/MapTaskExecutor.java#L130
It will affect all Reader.requestDynamicSplit calls including Beam IOs, user
IOs, shuffle reader, etc. One could not exclude the possibility that in some
case some logic may rely on the current behavior (no progress update sent on
split error). And I ended up with a limited scope of change where suppress this
particular error is known to be fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
