BasPH opened a new pull request, #28827: URL: https://github.com/apache/airflow/pull/28827
This PR adds the option to wait for completion with the EmrCreateJobFlowOperator. It includes: * The implementation of a `waiter` to wait for WAITING or TERMINATED state. * A few consistency fixes here and there. AWS writes "job flow" (lowercase) so I adhered to that. * Also implemented an `on_kill` method. Moved EMR hook creation & caching to a `cached_property` because the hook is called from multiple methods. * I don't think a default max. runtime of 25 minutes is desirable, I've had many projects which ran EMR jobs longer than that. I allowed `None` for the `waiter.countdown` argument type to represent waiting for infinity (or until the Airflow task times out based on `execution_timeout`). Defaulted the `EmrCreateJobFlowOperator.waiter_countdown` to `None`. Checked all other usages of `waiter_countdown` and they all default to `25 * 60` so no breaking changes. --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information. In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
