[ https://issues.apache.org/jira/browse/AIRFLOW-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431308#comment-16431308 ]
John Arnold commented on AIRFLOW-2156: -------------------------------------- This seems like a pretty easy candidate for multiprocessing. > Parallelize Celery Executor > --------------------------- > > Key: AIRFLOW-2156 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2156 > Project: Apache Airflow > Issue Type: Improvement > Components: celery > Reporter: Dan Davydov > Assignee: Dan Davydov > Priority: Major > > The CeleryExecutor doesn't currently support parallel execution to check task > states since Celery does not support this. This can greatly slow down the > Scheduler loops since each request to check a task's state is a network > request. > > The Celery Executor should parallelize these requests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)