Vivien Morlet created AIRFLOW-5516:
--------------------------------------
Summary: Webserver crashes when there are too many dags or tasks
to handle
Key: AIRFLOW-5516
URL: https://issues.apache.org/jira/browse/AIRFLOW-5516
Project: Apache Airflow
Issue Type: Bug
Components: DAG, ui, webserver
Affects Versions: 1.10.2, 1.10.1, 1.10.0, 1.9.0, 1.8.2, 1.8.1
Environment: Google Cloud Composer
Reporter: Vivien Morlet
Attachments: ServerError502.png, badGateway.png
Hello everyone,
I am using Apache Airflow with the orchestration service Composer of Google
Cloud.
I have an issue with the UI Webserver. I noticed that the webserver crashes
when there are too many DAGs or tasks to handle; by "crashes" I mean that the
workers are still handling the tasks and DAGs are triggered but the Webserver
throws 502 errors (see related screenshots).
I have approximately 50 DAGs that Airflow parses and there are for some DAGs
sometimes thousands of tasks to handle and to schedule.
As I said, the workers are still doing their jobs but this is totally
unconvenient (understand impossible) to handle DAGs (e.g trigger dags, show
graphs etc.) without the UI Webserver especially since the service Composer of
Google Cloud doesn't give a total access to every tools of Airflow.
Theoretically Airflow can handle an infinity of tasks but visibly the webserver
suffers.
What can I do with this issue?
Thanks for your help, I hope that we will be able to fix this issue soon.
Regards,
Bibimorlet
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
