Joar Wandborg created BEAM-5529:
-----------------------------------
Summary: Dataflow runner raises AssertionError if job takes > 50
seconds to go from PENDING to RUNNING
Key: BEAM-5529
URL: https://issues.apache.org/jira/browse/BEAM-5529
Project: Beam
Issue Type: Bug
Components: sdk-py-core
Affects Versions: 2.6.0
Reporter: Joar Wandborg
Assignee: Ahmet Altay
To reproduce:
- Try to run a Pipeline that takes longer than 50 seconds to go from
JOB_STATE_PENDING to JOB_STATE_RUNNING using "with Pipeline() as pipeline:"
(automatically executes {{pipeline.run().wait_until_finish()}} via
{{Pipeline.__exit__}}) on Dataflow.
- Your script will raise "AssertionError: Job did not reach to a terminal
state after waiting indefinitely." (from
[https://github.com/apache/beam/blob/766a1dc5b6523a0b7a3ce0a9ab78901cf17c9013/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L1152-L1153])
I think the issue is that the block at
[https://github.com/apache/beam/blob/766a1dc5b6523a0b7a3ce0a9ab78901cf17c9013/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L147-L160]
unintentionally exits if it takes > 50 seconds for a job to go from
JOB_STATE_PENDING to JOB_STATE_RUNNING.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
