[
https://issues.apache.org/jira/browse/BEAM-7628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16878212#comment-16878212
]
Kenneth Knowles commented on BEAM-7628:
---------------------------------------
What version of Beam is this?
[~tvalentyn] can you add more information?
The 499 is the result of some layer giving up on retries/time out.
> Retry createJob requests in Dataflow Runner for retriable errors.
> -----------------------------------------------------------------
>
> Key: BEAM-7628
> URL: https://issues.apache.org/jira/browse/BEAM-7628
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow
> Reporter: Valentyn Tymofieiev
> Priority: Major
>
> When Dataflow Runner is sending a job for remote execution, such requests in
> rare cases might fail with retriable errors. Dataflow Runner could recognize
> a class of retriable errors and attempt to resubmit the job again when such
> errors are encountered. Sample retriable error encountered by Beam Java SDK:
> ```
> java.lang.RuntimeException: Failed to create a workflow job: The operation
> was cancelled.
> 11:32:14 at
> org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:869)
> 11:32:14 at
> org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:178)
> 11:32:14 at org.apache.beam.sdk.Pipeline.run(Pipeline.java:313)
> 11:32:14 at org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
> ...
> 11:32:14 Caused by:
> com.google.api.client.googleapis.json.GoogleJsonResponseException: 499 Client
> Closed Request
> 11:32:14 {
> 11:32:14 "code" : 499,
> 11:32:14 "errors" : [ {
> 11:32:14 "domain" : "global",
> 11:32:14 "message" : "The operation was cancelled.",
> 11:32:14 "reason" : "backendError"
> 11:32:14 } ],
> 11:32:14 "message" : "The operation was cancelled.",
> 11:32:14 "status" : "CANCELLED"
> 11:32:14 }
> 11:32:14 at
> com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
> 11:32:14 at
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
> 11:32:14 at
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
> 11:32:14 at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321)
> 11:32:14 at
> com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1067)
> 11:32:14 at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
> 11:32:14 at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
> 11:32:14 at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
> 11:32:14 at
> org.apache.beam.runners.dataflow.DataflowClient.createJob(DataflowClient.java:61)
> 11:32:14 at
> org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:855)
> 11:32:14 ... 41 more'
> ```
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
