[
https://issues.apache.org/jira/browse/AIRFLOW-774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16111850#comment-16111850
]
Ace Haidrey commented on AIRFLOW-774:
-------------------------------------
Got it. That's why I'm not seeing the errors that you saw, since my version is
1.8.1. But I may try taking a stab at correcting these stats..
We are using [statsd_exporter|https://github.com/prometheus/statsd_exporter]
project with prometheus to monitor these instances and another issue is the
labels for the stats in Airflow have underscores vs dots to split up, but they
need to be dot separated so we will try changing that as well.
The statsd_exporter can be configured to translate specific dot-separated
StatsD metrics into labeled Prometheus metrics via a simple mapping language.
That's an aside but just giving a background of what we're trying to accomplish!
> dagbag_size/collect_dags/dagbag_import_errors stats incorrect
> -------------------------------------------------------------
>
> Key: AIRFLOW-774
> URL: https://issues.apache.org/jira/browse/AIRFLOW-774
> Project: Apache Airflow
> Issue Type: Bug
> Components: logging
> Reporter: Dan Davydov
>
> After the multiprocessor change was made (dag folders are processed in
> parallel), the number of dags reported by airflow is for each of these
> subprocesses which is inaccurate, and potentially orders of magnitude less
> than the actual number of dags. These individual processes stats should be
> aggregated. The collect_dags/dagbag_import_errors stats should also be fixed
> (time it takes to parse the dags).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
