suhanovv opened a new issue #17513: URL: https://github.com/apache/airflow/issues/17513
**Apache Airflow version**: 2.1.2 **Apache Airflow Provider versions** (please include all providers that are relevant to your bug): 2.0 apache-airflow-providers-apache-hive==2.0.0 apache-airflow-providers-celery==2.0.0 apache-airflow-providers-cncf-kubernetes==1.2.0 apache-airflow-providers-docker==2.0.0 apache-airflow-providers-elasticsearch==1.0.4 apache-airflow-providers-ftp==2.0.0 apache-airflow-providers-imap==2.0.0 apache-airflow-providers-jdbc==2.0.0 apache-airflow-providers-microsoft-mssql==2.0.0 apache-airflow-providers-mysql==2.0.0 apache-airflow-providers-oracle==2.0.0 apache-airflow-providers-postgres==2.0.0 apache-airflow-providers-sqlite==2.0.0 apache-airflow-providers-ssh==2.0.0 **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): 1.17 **Environment**: - **Install tools**: https://github.com/prometheus/statsd_exporter **What happened**: We are using an [statsd_exporter](https://github.com/prometheus/statsd_exporter) to store statsd metrics in prometheus. And we came across strange behavior, the metric dag_processing.last_duration. <dag_file> for different dags is drawn with holes at random intervals.    **What you expected to happen**: Metrics should be sent with the frequency specified in the config in the AIRFLOV__SHEDULER__PRINT_STATS_INTERVAL parameter and the default value is 30, and this happens in the[ _log_file_processing_stats method](https://github.com/apache/airflow/blob/2fea4cdceaa12b3ac13f24eeb383af624aacb2e7/airflow/dag_processing/manager.py#L696), the problem is that the initial time is taken using the [get_start_time](https://github.com/apache/airflow/blob/2fea4cdceaa12b3ac13f24eeb383af624aacb2e7/airflow/dag_processing/manager.py#L827) function which looks only at the probability of active processes that some of the dags will have time to be processed in 30 seconds and removed from self._processors [file_path] and thus the metrics for them will not be sent to the statsd. While for outputting to the log, [lasttime](https://github.com/apache/airflow/blob/2fea4cdceaa12b3ac13f24eeb383af624aacb2e7/airflow/dag_processing/manager.py#L783) is used in which information on the last processing is stored an d from which it would be nice to send metrics to the statistics. **How to reproduce it**: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
