Gary Harpaz created AIRFLOW-5818:
------------------------------------
Summary: Very bad webserver performance when defining many dags
with many operators
Key: AIRFLOW-5818
URL: https://issues.apache.org/jira/browse/AIRFLOW-5818
Project: Apache Airflow
Issue Type: Bug
Components: webserver
Affects Versions: 1.10.5, 1.10.4, 1.10.3, 1.10.2, 1.10.1, 1.10.0, 1.9.0
Reporter: Gary Harpaz
Attachments: dup_dags.py, my_dag.template
In my scenario I have defined 500 dags, each dag has approximately 1500
operators.
This makes webserver impossible to work with even when all dags are paused and
nothing is running. The cpu spikes all the time and webserver consumes huge
amounts of memory for no reason.
To reproduce this use the attched my_dag.template file and duplicate it using
the attached dup_dags.py script.
The root cause of this issue is that dagbag will load all dags into memory
which takes huge cpu and memory unnecessarily.
I have already fixed this in:
[https://github.com/gary-harpaz/airflow/tree/improve-performance]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
