enewnham opened a new issue, #45651: URL: https://github.com/apache/airflow/issues/45651
### Apache Airflow version 2.10.4 ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? The EmrCreateJobFlowOperator returns early when `wait_for_completion=True` due improperly treating `Running` as `success` in the acceptor. When a step is running that doesn't necessarily mean the job is complete, only waiting for new jobs or terminated should the cluster be considered idle and the job completed. This was introduced in https://github.com/apache/airflow/pull/37667, https://github.com/apache/airflow/pull/37667/files#diff-f4ade57b13bf58042c4171d225b9b28c7610072f8b33b722bb138db900feb7fcR65-R70 I added some printf debugging to find this issue ``` [2025-01-14, 17:22:34 UTC] {emr.py:706} INFO - Job flow with id j-YEXH6CUHAEDF created [2025-01-14, 17:22:34 UTC] {emr.py:737} INFO - job_flow_waiting [2025-01-14, 17:22:34 UTC] {waiter.py:394} INFO - here: wait [2025-01-14, 17:23:34 UTC] {waiter.py:394} INFO - here: wait [2025-01-14, 17:24:35 UTC] {waiter.py:394} INFO - here: wait [2025-01-14, 17:25:35 UTC] {waiter.py:369} INFO - here: current_state=success ```  ### What you think should happen instead? The EmrCreateJobFlowOperator should continue to wait_for_completion while there are steps still running in the EMR cluster ### How to reproduce Use the EmrCreateJobFlowOperator to create a cluster with one or more steps. Optionally set `"KeepJobFlowAliveWhenNoSteps": False,` ### Operating System Amazon Linux ### Versions of Apache Airflow Providers _No response_ ### Deployment Amazon (AWS) MWAA ### Deployment details _No response_ ### Anything else? Every time ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
