ephraimbuddy opened a new pull request #17819: URL: https://github.com/apache/airflow/pull/17819
Task callbacks in scheduler are sent to DAG file processor to process and are quite problematic. Most times, task instances that have retries are not retried up to the number required, see #16625. Also, task instances get stuck in up-for-retry state or in queued state which led to #15929. I believe this happens because the DAG file processor dies and a new one is created, which may not run the task callbacks. This PR fixes this by running the callbacks right when they fail instead of passing them to DAG file processor --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information. In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
