Elad created AIRFLOW-5328:
-----------------------------
Summary: EmrCreateJobFlowOperator success when EMR isn't ready
Key: AIRFLOW-5328
URL: https://issues.apache.org/jira/browse/AIRFLOW-5328
Project: Apache Airflow
Issue Type: Bug
Components: aws
Affects Versions: 1.10.4
Reporter: Elad
EmrCreateJobFlowOperator returns almost instantly.
It shouldn't return until the Emr cluster id ready.
It doesn't make sense to submit tasks to EMR when the cluster isn't ready.
Currently, tasks are submitted and waiting until the cluster will finish its
initialization.
This can be checked with Waiter:
[https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/emr.html]
something like:
{code:java}
waiter =
client.get_waiter('cluster_running').wait(ClusterId=response['JobFlowId'],
WaiterConfig={'Delay': 32, 'MaxAttempts': 60}){code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
