Dan Davydov created AIRFLOW-1311:
------------------------------------
Summary: Improve Webserver Load Time For Large DAGs
Key: AIRFLOW-1311
URL: https://issues.apache.org/jira/browse/AIRFLOW-1311
Project: Apache Airflow
Issue Type: Bug
Components: webserver
Reporter: Dan Davydov
Large DAGs can take an extremely long time to load in the Airflow UI
(minutes/timeout).
The fixes are as follows:
1. Lazy load DAGs (load up to a certain # of tasks by default, prioritizing
tasks by their depth, and allow users to expand sections for these DAGs,
ideally prefetch deeper tasks once the initial set of tasks has rendered )
2. Identify bottlenecks/performance issues in both the frontend/backend for
rendering DAGs on the webserver and fix them. Airflow should be more performant
for displaying DAGs that are somewhat large, e.g. DAGs that have up to 500
nodes and 2000 edges (dependencies from one task to another) should render
within a couple of seconds.
3. Make DAG loading asynchronous in the UI (once the top-level tasks have
loaded display them immediately). We might not want to do this as users might
try to click something only to have the UI change from underneath them
[~saguziel]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
