ashb commented on a change in pull request #5743: [AIRFLOW-5088][AIP-24]
Persisting serialized DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#discussion_r312813200
##########
File path: airflow/models/dagbag.py
##########
@@ -416,6 +449,27 @@ def collect_dags(
format(dag_names),
file_stat.duration)
+ def collect_dags_from_db(self):
+ """Collects DAGs from database."""
+ start_dttm = timezone.utcnow()
+ self.log.info("Filling up the DagBag from database")
+
+ # The dagbag contains all rows in serialized_dag table. Deleted DAGs
are deleted
+ # from the table by the scheduler job.
+ self.dags = SerializedDagModel.read_all_dags()
+
+ # Adds subdags.
+ # DAG post-pcocessing steps such as self.bag_dag and croniter are not
needed as
+ # they are done by scheduler before serialization.
+ subdags = {}
+ for dag in self.dags.values():
+ for subdag in dag.subdags:
+ subdags[subdag.dag_id] = subdag
+ self.dags.update(subdags)
+
+ Stats.timing('collect_dags', timezone.utcnow() - start_dttm)
Review comment:
I don't think we should be emitting this metric here -- it will get mixed in
with the same metric from the scheduler and will have a different mean/std dev
etc. This should be a different metric.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
