[
https://issues.apache.org/jira/browse/AIRFLOW-2615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16512059#comment-16512059
]
Kevin Yang commented on AIRFLOW-2615:
-------------------------------------
Adding a little bit context here: Airbnb has ~2000 DAG file in our centralized
DAG repo and it takes a long time to parse the entire repo, this extra app
creation is basically doubling the time we need to refresh webserver worker.
> Webserver not using cached app
> ------------------------------
>
> Key: AIRFLOW-2615
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2615
> Project: Apache Airflow
> Issue Type: Bug
> Reporter: Kevin Yang
> Priority: Major
>
> From what I can tell, the app cached
> [here|https://github.com/apache/incubator-airflow/blob/master/airflow/bin/cli.py#L790]
> attempt to cache the app for later use-likely to be for the expensive
> DagBag() creation. Before I dive into the webserver parsing everything in one
> process problem, I was hoping this cached app would save me sometime. However
> it seems to me that every subprocess spun up by gunicorn is trying to create
> the DagBag() right after they've been created--make sense to me since we
> didn't share the cached app to the subprocess( doubt we can). If what I
> observed is true, why do we cache the app at all?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
