noamst-monday opened a new issue, #45285: URL: https://github.com/apache/airflow/issues/45285
### Apache Airflow version 2.10.4 ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? Hello! We are running Airflow 2.10.4 on EKS version 1.30.6. I set up statsd metric collection as described [here](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/metrics.html). Using datadog agent running as a daemonset on each node to collect the metrics. I am able to receive and search all metrics successfully in datadog, but task.scheduled_duration is very rarely sent, and when it's sent the numbers and tagging don't make sense to me. For example, over the last week I only have a single datapoint at 3.26k, and it seems like it's tagged with several task_ids and dag_ids. ### What you think should happen instead? I expect to receive the metric every time a task is scheduled, and tagged correctly with only the relevant task_id and dag_id. ### How to reproduce Airflow 2.10.4 on EKS 1.30, using official helm chart version 1.13.1. Airflow metrics configuration: [metrics] metrics_use_pattern_match = False metrics_allow_list = metrics_block_list = statsd_on = true statsd_host = 10.128.31.107 statsd_port = 8125 statsd_prefix = airflow stat_name_handler = statsd_datadog_enabled = True statsd_datadog_tags = project:dbt statsd_datadog_metrics_tags = True # statsd_custom_client_path = statsd_disabled_tags = job_id,run_id statsd_influxdb_enabled = False ### Operating System Debian GNU/Linux 12 (bookworm) ### Versions of Apache Airflow Providers apache-airflow==2.10.4 apache-airflow-providers-amazon==9.1.0 apache-airflow-providers-celery==3.8.5 apache-airflow-providers-cncf-kubernetes==10.0.1 apache-airflow-providers-common-compat==1.2.2 apache-airflow-providers-common-io==1.4.2 apache-airflow-providers-common-sql==1.20.0 apache-airflow-providers-dbt-cloud==3.11.2 apache-airflow-providers-docker==3.14.1 apache-airflow-providers-elasticsearch==5.5.3 apache-airflow-providers-fab==1.5.1 apache-airflow-providers-ftp==3.11.1 apache-airflow-providers-google==11.0.0 apache-airflow-providers-grpc==3.6.0 apache-airflow-providers-hashicorp==3.8.0 apache-airflow-providers-http==4.13.3 apache-airflow-providers-imap==3.7.0 apache-airflow-providers-microsoft-azure==11.1.0 apache-airflow-providers-mysql==5.7.4 apache-airflow-providers-odbc==4.8.1 apache-airflow-providers-openlineage==1.14.0 apache-airflow-providers-postgres==5.14.0 apache-airflow-providers-redis==3.8.0 apache-airflow-providers-sendgrid==3.6.0 apache-airflow-providers-sftp==4.11.1 apache-airflow-providers-slack==8.9.2 apache-airflow-providers-smtp==1.8.1 apache-airflow-providers-snowflake==5.8.1 apache-airflow-providers-sqlite==3.9.1 apache-airflow-providers-ssh==3.14.0 ### Deployment Official Apache Airflow Helm Chart ### Deployment details Airflow 2.10.4 on EKS 1.30, using official helm chart version 1.13.1. Datadog version - `datadog/agent:7.56.2` Airflow metrics configuration: ``` [metrics] metrics_use_pattern_match = False metrics_allow_list = metrics_block_list = statsd_on = true statsd_host = 10.128.31.107 statsd_port = 8125 statsd_prefix = airflow stat_name_handler = statsd_datadog_enabled = True statsd_datadog_tags = project:dbt statsd_datadog_metrics_tags = True # statsd_custom_client_path = statsd_disabled_tags = job_id,run_id statsd_influxdb_enabled = False ``` ### Anything else? Thank you, and please let me know if there are any additional details I can provide to help triage or reproduce this issue. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
