[
https://issues.apache.org/jira/browse/BEAM-5529?focusedWorklogId=151419&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151419
]
ASF GitHub Bot logged work on BEAM-5529:
----------------------------------------
Author: ASF GitHub Bot
Created on: 04/Oct/18 22:53
Start Date: 04/Oct/18 22:53
Worklog Time Spent: 10m
Work Description: aaltay closed pull request #6535: [BEAM-5529] Before
starting the final countdown, check that job is not in pre-running state
URL: https://github.com/apache/beam/pull/6535
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
index 868d264a9d9..1acd3488524 100644
--- a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
+++ b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
@@ -155,9 +155,15 @@ def rank_error(msg):
or str(response.currentState) == 'JOB_STATE_UPDATED'
or str(response.currentState) == 'JOB_STATE_DRAINED'):
break
- # The job has failed; ensure we see any final error messages.
- sleep_secs = 1.0 # poll faster during the final countdown
- final_countdown_timer_secs -= sleep_secs
+
+ # Check that job is in a post-preparation state before starting the
+ # final countdown.
+ if (str(response.currentState) not in (
+ 'JOB_STATE_PENDING', 'JOB_STATE_QUEUED')):
+ # The job has failed; ensure we see any final error messages.
+ sleep_secs = 1.0 # poll faster during the final countdown
+ final_countdown_timer_secs -= sleep_secs
+
time.sleep(sleep_secs)
# Get all messages since beginning of the job run or since last message.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 151419)
Time Spent: 3h 10m (was: 3h)
> Dataflow runner raises AssertionError if job takes > 50 seconds to go from
> PENDING to RUNNING
> ---------------------------------------------------------------------------------------------
>
> Key: BEAM-5529
> URL: https://issues.apache.org/jira/browse/BEAM-5529
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Affects Versions: 2.6.0
> Reporter: Joar Wandborg
> Assignee: Ahmet Altay
> Priority: Major
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> To reproduce:
> - Try to run a Pipeline that takes longer than 50 seconds to go from
> JOB_STATE_PENDING to JOB_STATE_RUNNING using "with Pipeline() as pipeline:"
> (automatically executes {{pipeline.run().wait_until_finish()}} via
> {{Pipeline.__exit__}}) on Dataflow.
> - Your script will raise "AssertionError: Job did not reach to a terminal
> state after waiting indefinitely." (from
> [https://github.com/apache/beam/blob/766a1dc5b6523a0b7a3ce0a9ab78901cf17c9013/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L1152-L1153])
> I think the issue is that the block at
> [https://github.com/apache/beam/blob/766a1dc5b6523a0b7a3ce0a9ab78901cf17c9013/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L147-L160]
> unintentionally exits if it takes > 50 seconds for a job to go from
> JOB_STATE_PENDING to JOB_STATE_RUNNING.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
