msumit commented on issue #5594: [AIRFLOW-4924] Loading DAGs asynchronously in Airflow webserver URL: https://github.com/apache/airflow/pull/5594#issuecomment-511755626 Thanks for much-needed improvement in webserver @coufon. I would like to understand more about the workings of this enhancement: - So each gunicorn worker going to launch a separate thread to parse DAGs asynchronously, and not a single background process going to do the DAG parsing for all the workers. - The pickled DAG object won't be stored into DB and just used for thread communication. - As of now each gunicorn worker might have diff copy of DAGs (due to changes) and cause confusion on UI, something which isn't going to solve with this change. - How it going to behave if DAG parsing time for each thread is more than the configured `collect_dags_interval` time.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
