[
https://issues.apache.org/jira/browse/AIRFLOW-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16548352#comment-16548352
]
Kevin Yang commented on AIRFLOW-2762:
-------------------------------------
[~ashb] Good idea. Though I am a bit concerned about the parsing time--we have
a couple framework DAGs that takes tens of seconds to parse. I think in this
case cache beforehand during start up may even be better than cache lazily.
This might also create two sources for webserver to find DAG and potentially
create inconsistency within the webserver if the files on the scheduler and
webservers are not synced. I think to parse the DAG into simple DAG would be a
relatively safer way to approach this.
> Parallelize DAG parsing in webserver
> ------------------------------------
>
> Key: AIRFLOW-2762
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2762
> Project: Apache Airflow
> Issue Type: Improvement
> Reporter: Kevin Yang
> Priority: Major
>
> Currently the webserver parses DagBag in a single thread fashion and causes
> the start up time to be slow when we have large # of DAG files. Webservers
> should not need the actual DAG object and this should be parallelized.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
