[
https://issues.apache.org/jira/browse/AIRFLOW-5516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor resolved AIRFLOW-5516.
----------------------------------------
Resolution: Duplicate
We are working on this already via
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persistence+in+DB+using+JSON+for+Airflow+Webserver+and+%28optional%29+Scheduler
and hope to have it released in 1.10.6 (or .7 if we decide .6 needs to be a
bug fix release first.)
> Webserver crashes when there are too many dags or tasks to handle
> -----------------------------------------------------------------
>
> Key: AIRFLOW-5516
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5516
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG, ui, webserver
> Affects Versions: 1.8.1, 1.8.2, 1.9.0, 1.10.0, 1.10.1, 1.10.2
> Environment: Google Cloud Composer
> Reporter: Vivien Morlet
> Priority: Major
> Labels: DAG, Error, UI, tasks, webserver
> Attachments: ServerError502.png, badGateway.png
>
>
> Hello everyone,
>
> I am using Apache Airflow with the orchestration service Composer of Google
> Cloud.
> I have an issue with the UI Webserver. I noticed that the webserver crashes
> when there are too many DAGs or tasks to handle; by "crashes" I mean that the
> workers are still handling the tasks and DAGs are triggered but the Webserver
> throws 502 errors (see related screenshots).
>
> I have approximately 50 DAGs that Airflow parses and there are for some DAGs
> sometimes thousands of tasks to handle and to schedule.
>
> As I said, the workers are still doing their jobs but this is totally
> unconvenient (understand impossible) to handle DAGs (e.g trigger dags, show
> graphs etc.) without the UI Webserver especially since the service Composer
> of Google Cloud doesn't give a total access to every tools of Airflow.
>
> Theoretically Airflow can handle an infinity of tasks but visibly the
> webserver suffers.
> What can I do with this issue?
>
> Thanks for your help, I hope that we will be able to fix this issue soon.
> Regards,
> Bibimorlet
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
