adambaker opened a new issue, #28704: URL: https://github.com/apache/airflow/issues/28704
### Apache Airflow Provider(s) amazon ### Versions of Apache Airflow Providers 6.2.0 ### Apache Airflow version 2.5.0 ### Operating System CentOS 7 ### Deployment Other Docker-based deployment ### Deployment details _No response_ ### What happened Upon upgrading to airflow 2.5.0, EMR Add Steps Operator now: 1. Succeeds at adding a step to our emr job. 2. tries to wait for the job to complete 3. times out, retries, and duplicates the steps on our emr cluster. 4. times out, and fails. The cause seems to be an error in the operator: https://github.com/apache/airflow/pull/27286/files#diff-94612f7f88a5f95ca9e3f23051f39f1e0b6a4cadac1f533615db5480b379ed34R115 Instead of passing `wait_for_completion=True` to the emr hook method `add_job_flow_steps`, it should probably pass `self.wait_for_completion`. Also, currently `self.wait_for_completion` seem to not be used anywhere in the project. ### What you think should happen instead The task should succeed once the EMR steps are added and move on to the next task in the dag. ### How to reproduce _No response_ ### Anything else _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
