johnson-jay-l opened a new issue, #32271: URL: https://github.com/apache/airflow/issues/32271
### Apache Airflow version Other Airflow 2 version (please specify below) ### What happened **Airflow version: 2.3.3** We integrate airflow metrics with datadog, and there should be "counter" events from [this doc](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/metrics.html#counters) that post to `ti.finish.<dag_id>.<task_id>.<state>` We are not finding any events for the `upstream_failed` state. We need the ability to monitor and alert on that state because it leads to SLA failure alarms, and we'd like to find out about the issue as it happens so that corrective action can be taken, instead of waiting for the alarm to fire. In other words, we want to alert on the `upstream_failed` state, instead of on the absence of success after X period of time elapses. From what I can tell, the state of `upstream_failed` is not included in the failure callbacks. ### What you think should happen instead `upstream_failed` ti state events should post to `ti.finish.<dag_id>.<task_id>.<state>` ### How to reproduce 1. Create a dag `test_dag` with 2 empty tasks `task_1` and `task_2`, and task dependencies of `task_1 >> task_2` 2. Unpause the dag and fail `task_1` 3. Observe that `task_2` is in state `upstream_failed` 4. Observe that no metrics are emitted for the counter value of `ti.finish.test_dag.task_2.upstream_failed` ### Operating System Debian GNU/Linux 10 (buster) ### Versions of Apache Airflow Providers snippit from requirements.txt: ``` apache-airflow[statsd]==2.3.3 # via # -r requirements.in # airflow-provider-fivetran # apache-airflow-providers-amazon # apache-airflow-providers-cncf-kubernetes # apache-airflow-providers-postgres # apache-airflow-providers-salesforce # apache-airflow-providers-slack apache-airflow-providers-amazon==6.0.0 # via -r requirements.in apache-airflow-providers-cncf-kubernetes==2.1.0 # via -r requirements.in apache-airflow-providers-common-sql==1.4.0 # via # -r requirements.in # apache-airflow-providers-amazon apache-airflow-providers-ftp==2.0.1 # via apache-airflow apache-airflow-providers-http==2.0.1 # via # apache-airflow # apache-airflow-providers-slack apache-airflow-providers-imap==2.0.1 # via apache-airflow apache-airflow-providers-postgres==2.2.0 # via -r requirements.in apache-airflow-providers-salesforce==3.0.0 # via -r requirements.in apache-airflow-providers-slack[http]==4.0.1 # via -r requirements.in apache-airflow-providers-sqlite==2.0.1 # via apache-airflow ``` ### Deployment Official Apache Airflow Helm Chart ### Deployment details n/a ### Anything else n/a ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
