potiuk commented on issue #5594: [AIRFLOW-4924] Loading DAGs asynchronously in Airflow webserver URL: https://github.com/apache/airflow/pull/5594#issuecomment-511661597 And I would not worry about multiprocessing queue problems in this case. In this case we have simple Producer -> Consumer case and we use it in the simplest possible way -> put() in producer and get() in consumer. The problems with multiprocessing queues that we experienced were related to rather complex pattern of using the queues where we had several queues synchronising back-forth and we were using length of the queue as indication of whether there is anything left for processing. I would however possibly think about increasing the maxsize of the Queue (will leave a comment there) to increase parallelism.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
