We've noticed this with celery, relating to this https://github.com/celery/celery/issues/3765
We also use `-n 5` option on the scheduler so it restarts every 5 runs, which will reset all queued tasks. Best, Alex On Thu, Jun 1, 2017 at 2:18 PM, Josef Samanek <[email protected]> wrote: > Hi! > > We have a problem with our airflow. Sometimes, several tasks get queued > but they never get run and remain in Queud state forever. Other tasks from > the same schedule interval run. And next schedule interval runs normally > too. But these several tasks remain queued. > > We are using Airflow 1.8.1. Currently with CeleryExecutor and redis, but > we had the same problem with LocalExecutor as well (actually switching to > Celery helped quite a bit, the problem now happens way less often, but > still it happens). We have 18 DAGs total, 13 active. Some have just 1-2 > tasks, but some are more complex, like 8 tasks or so and with upstreams. > There are also ExternalTaskSensor tasks used. > > I tried playing around with DAG configurations (limiting concurrency, > max_active_runs, ...), tried switching off some DAGs completely (not all > but most) etc., so far nothing helped. Right now, I am not really sure, > what else to try to identify a solve the issue. > > I am getting a bit desperate, so I would really appreciate any help with > this. Thank you all in advance! > > Joe >
