[
https://issues.apache.org/jira/browse/AIRFLOW-774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16936595#comment-16936595
]
ASF subversion and git services commented on AIRFLOW-774:
---------------------------------------------------------
Commit 5f9ab7a1d5cd4540e953005d43898be08ed56d60 in airflow's branch
refs/heads/master from Ash Berlin-Taylor
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=5f9ab7a ]
[AIRFLOW-774] Fix long-broken DAG parsing Statsd metrics (#6157)
Since we switched to using sub-processes to parse the DAG files sometime
back in 2016(!) the metrics we have been emitting about dag bag size and
parsing have been incorrect.
We have also been emitting metrics from the webserver which is going to
be become wrong as we move towards a stateless webserver.
To fix both of these issues I have stopped emitting the metrics from
models.DagBag and only emit them from inside the
DagFileProcessorManager.
(There was also a bug in the `dag.loading-duration.*` we were emitting
from the DagBag code where the "dag_file" part of that metric was empty.
I have fixed that even though I have now deprecated that metric. The
webserver was emitting the right metric though so many people wouldn't
notice)
> dagbag_size/collect_dags/dagbag_import_errors stats incorrect
> -------------------------------------------------------------
>
> Key: AIRFLOW-774
> URL: https://issues.apache.org/jira/browse/AIRFLOW-774
> Project: Apache Airflow
> Issue Type: Bug
> Components: logging
> Reporter: Dan Davydov
> Assignee: Ash Berlin-Taylor
> Priority: Major
>
> After the multiprocessor change was made (dag folders are processed in
> parallel), the number of dags reported by airflow is for each of these
> subprocesses which is inaccurate, and potentially orders of magnitude less
> than the actual number of dags. These individual processes stats should be
> aggregated. The collect_dags/dagbag_import_errors stats should also be fixed
> (time it takes to parse the dags).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
