Hi! We have a problem with our airflow. Sometimes, several tasks get queued but they never get run and remain in Queud state forever. Other tasks from the same schedule interval run. And next schedule interval runs normally too. But these several tasks remain queued.
We are using Airflow 1.8.1. Currently with CeleryExecutor and redis, but we had the same problem with LocalExecutor as well (actually switching to Celery helped quite a bit, the problem now happens way less often, but still it happens). We have 18 DAGs total, 13 active. Some have just 1-2 tasks, but some are more complex, like 8 tasks or so and with upstreams. There are also ExternalTaskSensor tasks used. I tried playing around with DAG configurations (limiting concurrency, max_active_runs, ...), tried switching off some DAGs completely (not all but most) etc., so far nothing helped. Right now, I am not really sure, what else to try to identify a solve the issue. I am getting a bit desperate, so I would really appreciate any help with this. Thank you all in advance! Joe
