[ 
https://issues.apache.org/jira/browse/BEAM-1187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov closed BEAM-1187.
----------------------------------
       Resolution: Fixed
    Fix Version/s: 2.3.0

This was fixed by 
https://github.com/apache/beam/commit/032c2d82167e4134d4777d9b941ec6f5e47c0d53

> GCP Transport not performing timed backoff after connection failure
> -------------------------------------------------------------------
>
>                 Key: BEAM-1187
>                 URL: https://issues.apache.org/jira/browse/BEAM-1187
>             Project: Beam
>          Issue Type: Bug
>          Components: io-java-gcp, runner-dataflow, sdk-java-core
>            Reporter: Luke Cwik
>            Priority: Minor
>             Fix For: 2.3.0
>
>
> The http request retries are failing and seemingly being immediately retried 
> if there is a connection exception. Note that below all the times are the 
> same, and also that we are logging too much. This seems to be related to the 
> interaction by the chaining http request initializer combining the Credential 
> initializer followed by the RetryHttpRequestInitializer. Also, note that we 
> never log "Request failed with IOException, will NOT retry" which implies 
> that the retry logic never made it to the RetryHttpRequestInitializer.
> Action items are:
> 1) Ensure that the RetryHttpRequestInitializer is used
> 2) Ensure that calls do backoff
> 3) PR/3430 -Reduce the logging to one terminal statement saying that we 
> retried X times and final failure was YYY-
> Dump of console output:
> Dec 20, 2016 9:12:20 AM 
> com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner fromOptions
> INFO: PipelineOptions.filesToStage was not specified. Defaulting to files 
> from the classpath: will stage 1 files. Enable logging at DEBUG level to see 
> which files will be staged.
> Dec 20, 2016 9:12:21 AM 
> com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner run
> INFO: Executing pipeline on the Dataflow Service, which will have billing 
> implications related to Google Compute Engine usage and other Google Cloud 
> Services.
> Dec 20, 2016 9:12:21 AM com.google.cloud.dataflow.sdk.util.PackageUtil 
> stageClasspathElements
> INFO: Uploading 1 files from PipelineOptions.filesToStage to staging location 
> to prepare for execution.
> Dec 20, 2016 9:12:21 AM com.google.cloud.dataflow.sdk.util.PackageUtil 
> stageClasspathElements
> INFO: Uploading PipelineOptions.filesToStage complete: 1 files newly 
> uploaded, 0 files cached
> Dec 20, 2016 9:12:22 AM com.google.api.client.http.HttpRequest execute
> WARNING: exception thrown while executing request
> java.net.ConnectException: Connection refused
>       at java.net.PlainSocketImpl.socketConnect(Native Method)
>       at 
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>       at 
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>       at 
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>       at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>       at java.net.Socket.connect(Socket.java:589)
>       at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
>       at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
>       at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
>       at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
>       at sun.net.www.http.HttpClient.New(HttpClient.java:308)
>       at sun.net.www.http.HttpClient.New(HttpClient.java:326)
>       at 
> sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1169)
>       at 
> sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1105)
>       at 
> sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:999)
>       at 
> sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:933)
>       at 
> sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1283)
>       at 
> sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
>       at 
> com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
>       at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)
>       at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
>       at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
>       at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
>       at 
> com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner.run(DataflowPipelineRunner.java:632)
>       at 
> com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner.run(DataflowPipelineRunner.java:201)
>       at com.google.cloud.dataflow.sdk.Pipeline.run(Pipeline.java:181)
>       at 
> com.google.cloud.dataflow.integration.NumbersStreaming.numbersStreamingFromPubsub(NumbersStreaming.java:378)
>       at 
> com.google.cloud.dataflow.integration.NumbersStreaming.main(NumbersStreaming.java:831)
> Dec 20, 2016 9:12:22 AM com.google.api.client.http.HttpRequest execute
> WARNING: exception thrown while executing request
> java.net.ConnectException: Connection refused
>       at java.net.PlainSocketImpl.socketConnect(Native Method)
>       at 
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>       at 
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>       at 
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>       at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>       at java.net.Socket.connect(Socket.java:589)
>       at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
>       at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
>       at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
>       at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
>       at sun.net.www.http.HttpClient.New(HttpClient.java:308)
>       at sun.net.www.http.HttpClient.New(HttpClient.java:326)
>       at 
> sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1169)
>       at 
> sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1105)
>       at 
> sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:999)
>       at 
> sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:933)
>       at 
> sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1283)
>       at 
> sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
>       at 
> com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
>       at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)
>       at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
>       at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
>       at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
>       at 
> com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner.run(DataflowPipelineRunner.java:632)
>       at 
> com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner.run(DataflowPipelineRunner.java:201)
>       at com.google.cloud.dataflow.sdk.Pipeline.run(Pipeline.java:181)
>       at 
> com.google.cloud.dataflow.integration.NumbersStreaming.numbersStreamingFromPubsub(NumbersStreaming.java:378)
>       at 
> com.google.cloud.dataflow.integration.NumbersStreaming.main(NumbersStreaming.java:831)
> Dec 20, 2016 9:12:22 AM com.google.api.client.http.HttpRequest execute
> WARNING: exception thrown while executing request
> java.net.ConnectException: Connection refused
>       at java.net.PlainSocketImpl.socketConnect(Native Method)
>       at 
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>       at 
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>       at 
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>       at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>       at java.net.Socket.connect(Socket.java:589)
>       at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
>       at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
>       at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
>       at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
>       at sun.net.www.http.HttpClient.New(HttpClient.java:308)
>       at sun.net.www.http.HttpClient.New(HttpClient.java:326)
>       at 
> sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1169)
>       at 
> sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1105)
>       at 
> sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:999)
>       at 
> sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:933)
>       at 
> sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1283)
>       at 
> sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
>       at 
> com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
>       at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)
>       at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
>       at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
>       at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
>       at 
> com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner.run(DataflowPipelineRunner.java:632)
>       at 
> com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner.run(DataflowPipelineRunner.java:201)
>       at com.google.cloud.dataflow.sdk.Pipeline.run(Pipeline.java:181)
>       at 
> com.google.cloud.dataflow.integration.NumbersStreaming.numbersStreamingFromPubsub(NumbersStreaming.java:378)
>       at 
> com.google.cloud.dataflow.integration.NumbersStreaming.main(NumbersStreaming.java:831)
> Dec 20, 2016 9:12:22 AM com.google.api.client.http.HttpRequest execute
> WARNING: exception thrown while executing request
> java.net.ConnectException: Connection refused
>       at java.net.PlainSocketImpl.socketConnect(Native Method)
>       at 
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>       at 
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>       at 
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>       at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>       at java.net.Socket.connect(Socket.java:589)
>       at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
>       at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
>       at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
>       at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
>       at sun.net.www.http.HttpClient.New(HttpClient.java:308)
>       at sun.net.www.http.HttpClient.New(HttpClient.java:326)
>       at 
> sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1169)
>       at 
> sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1105)
>       at 
> sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:999)
>       at 
> sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:933)
>       at 
> sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1283)
>       at 
> sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
>       at 
> com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
>       at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)
>       at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
>       at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
>       at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
>       at 
> com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner.run(DataflowPipelineRunner.java:632)
>       at 
> com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner.run(DataflowPipelineRunner.java:201)
>       at com.google.cloud.dataflow.sdk.Pipeline.run(Pipeline.java:181)
>       at 
> com.google.cloud.dataflow.integration.NumbersStreaming.numbersStreamingFromPubsub(NumbersStreaming.java:378)
>       at 
> com.google.cloud.dataflow.integration.NumbersStreaming.main(NumbersStreaming.java:831)
> Dec 20, 2016 9:12:22 AM com.google.api.client.http.HttpRequest execute
> WARNING: exception thrown while executing request
> java.net.ConnectException: Connection refused
>       at java.net.PlainSocketImpl.socketConnect(Native Method)
>       at 
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>       at 
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>       at 
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>       at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>       at java.net.Socket.connect(Socket.java:589)
>       at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
>       at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
>       at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
>       at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
>       at sun.net.www.http.HttpClient.New(HttpClient.java:308)
>       at sun.net.www.http.HttpClient.New(HttpClient.java:326)
>       at 
> sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1169)
>       at 
> sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1105)
>       at 
> sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:999)
>       at 
> sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:933)
>       at 
> sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1283)
>       at 
> sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
>       at 
> com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
>       at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)
>       at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
>       at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
>       at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
>       at 
> com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner.run(DataflowPipelineRunner.java:632)
>       at 
> com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner.run(DataflowPipelineRunner.java:201)
>       at com.google.cloud.dataflow.sdk.Pipeline.run(Pipeline.java:181)
>       at 
> com.google.cloud.dataflow.integration.NumbersStreaming.numbersStreamingFromPubsub(NumbersStreaming.java:378)
>       at 
> com.google.cloud.dataflow.integration.NumbersStreaming.main(NumbersStreaming.java:831)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to