[
https://issues.apache.org/jira/browse/AIRFLOW-5818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor resolved AIRFLOW-5818.
----------------------------------------
Fix Version/s: 1.10.7
Resolution: Fixed
I think this is fixed now in 1.10.7+ if you enable the serialized_dag option.
See
http://airflow.apache.org/docs/stable/dag-serialization.html?highlight=serialized
> Very bad webserver performance when defining many dags with many operators
> --------------------------------------------------------------------------
>
> Key: AIRFLOW-5818
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5818
> Project: Apache Airflow
> Issue Type: Bug
> Components: webserver
> Affects Versions: 1.9.0, 1.10.0, 1.10.1, 1.10.2, 1.10.3, 1.10.4, 1.10.5
> Reporter: Gary Harpaz
> Priority: Blocker
> Fix For: 1.10.7
>
> Attachments: Screenshot from 2019-11-03 17-39-43.png, dup_dags.py,
> my_dag.template
>
>
> In my scenario I have defined 500 dags, each dag has approximately 1500
> operators.
> This makes webserver impossible to work with even when all dags are paused
> and nothing is running. The cpu spikes all the time and webserver consumes
> huge amounts of memory for no reason.
> To reproduce this use the attched my_dag.template file and duplicate it using
> the attached dup_dags.py script.
>
> The root cause of this issue is that dagbag will load all dags into memory
> which takes huge cpu and memory unnecessarily.
> I have already fixed this in:
> [https://github.com/gary-harpaz/airflow/tree/improve-performance]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
