[
https://issues.apache.org/jira/browse/BEAM-7933?focusedWorklogId=321589&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-321589
]
ASF GitHub Bot logged work on BEAM-7933:
----------------------------------------
Author: ASF GitHub Bot
Created on: 01/Oct/19 22:21
Start Date: 01/Oct/19 22:21
Worklog Time Spent: 10m
Work Description: ecanzonieri commented on pull request #9673:
[BEAM-7933] Add job server request timeout (default to 60 seconds)
URL: https://github.com/apache/beam/pull/9673#discussion_r330302508
##########
File path: sdks/python/apache_beam/runners/portability/portable_runner.py
##########
@@ -251,7 +253,11 @@ def send_options_request(max_retries=5):
# This reports channel is READY but connections may fail
# Seems to be only an issue on Mac with port forwardings
return job_service.DescribePipelineOptions(
- beam_job_api_pb2.DescribePipelineOptionsRequest())
+ beam_job_api_pb2.DescribePipelineOptionsRequest(),
+ timeout=portable_options.job_server_timeout)
Review comment:
hmmm, I may be missing something but the timeout is never added to the
request. My understanding is that timeout itself is not a request parameter but
rather a grpc method call parameter. @ibzib can you point me to the code where
you see me adding the timeout to the request? Perhaps, I accidentally added it
to the wrong place.
I'm no grpc expert, but I've ran some manual tests on this feature:
1) run python driver without job server running and default timeout -> The
driver times out as expected after 60 seconds.
2) run python driver without job server running and change timeout -> The
driver times out as expected after the specified timeout.
3) run python driver with job server running but code changed to slow down
Prepare() -> The driver times out as expected.
4) run python driver with long running streaming job (Run() method) -> the
driver program never times out (note this is the expected behavior since the
driver is stuck waiting for completion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 321589)
Time Spent: 2h (was: 1h 50m)
> Adding timeout to JobServer grpc calls
> --------------------------------------
>
> Key: BEAM-7933
> URL: https://issues.apache.org/jira/browse/BEAM-7933
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
> Affects Versions: 2.14.0
> Reporter: Enrico Canzonieri
> Assignee: Enrico Canzonieri
> Priority: Minor
> Labels: portability
> Time Spent: 2h
> Remaining Estimate: 0h
>
> grpc calls to the JobServer from the Python SDK do not have timeouts. That
> means that the call to pipeline.run()could hang forever if the JobServer is
> not running (or failing to start).
> E.g.
> [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/portable_runner.py#L307]
> the call to Prepare() doesn't provide any timeout value and the same applies
> to other JobServer requests.
> As part of this ticket we could add a default timeout of 60 seconds as the
> default timeout for http client.
> Additionally, we could consider adding a --job-server-request-timeout to the
> [PortableOptions|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py#L805]
> class to be used in the JobServer interactions inside probable_runner.py.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
