zhaorui2022 opened a new issue, #69004:
URL: https://github.com/apache/airflow/issues/69004

   ### Description
   
   With Airflow 3's `Max Consecutive Failed DAG Runs` config, platform teams 
can automatically pause DAGs that repeatedly fail, preventing unnecessary 
resource consumption. However, once a DAG is auto-paused, there is currently no 
straightforward way for DAG owners to be notified.
   
   While richer notification mechanisms (such as callbacks) may require 
additional design discussion, emitting a StatsD metric when the scheduler 
automatically pauses a DAG would be a simple and useful first step. Users could 
leverage this metric to configure alerts through their existing monitoring 
systems and promptly notify DAG owners when their DAG has been auto-paused.
   
   ### Use case/motivation
   
   This metric would help users detect when a DAG has been automatically paused 
after exceeding the configured consecutive failure threshold.
   
   Today, Airflow emits metrics for individual DAG run failures, but those 
metrics do not make it easy to determine when a DAG has reached the failure 
threshold that triggers automatic pausing. Users would need to correlate 
consecutive failures themselves, which can be difficult and error-prone, 
especially across retries, restarts, or multiple monitoring systems.
   
   Emitting a dedicated metric when the scheduler automatically pauses a DAG 
would provide a clear signal that the configured threshold has been reached. 
Users could use this metric to trigger alerts, create dashboards, and respond 
promptly when a DAG stops being scheduled due to repeated failures.
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to