Depending on how fast you can clear down your queue, -n can be harmful and really stack up your celery queue. Keep an eye on your queue depth of you see a ton of messages about the task already having been run.
On Mon, Jun 5, 2017, 9:18 AM Josef Samanek <[email protected]> wrote: > Hey. Thanks for the answer. I previously also tried to run scheduler -n > 10, but it was back when I was still using LocalExecutor. And it did not > help. I have not yet tried to do it with CeleryExecutor, so I might. > > Still, I would prefer to find an actual solution for the underlying > problem, not just a workaround (eventhough a working workaround is also > appreciated). > > Best regards, > Joe > > On 2017-06-02 00:10 (+0200), Alex Guziel <[email protected]> > wrote: > > We've noticed this with celery, relating to this > > https://github.com/celery/celery/issues/3765 > > > > We also use `-n 5` option on the scheduler so it restarts every 5 runs, > > which will reset all queued tasks. > > > > Best, > > Alex > > > > On Thu, Jun 1, 2017 at 2:18 PM, Josef Samanek <[email protected]> > > wrote: > > > > > Hi! > > > > > > We have a problem with our airflow. Sometimes, several tasks get queued > > > but they never get run and remain in Queud state forever. Other tasks > from > > > the same schedule interval run. And next schedule interval runs > normally > > > too. But these several tasks remain queued. > > > > > > We are using Airflow 1.8.1. Currently with CeleryExecutor and redis, > but > > > we had the same problem with LocalExecutor as well (actually switching > to > > > Celery helped quite a bit, the problem now happens way less often, but > > > still it happens). We have 18 DAGs total, 13 active. Some have just 1-2 > > > tasks, but some are more complex, like 8 tasks or so and with > upstreams. > > > There are also ExternalTaskSensor tasks used. > > > > > > I tried playing around with DAG configurations (limiting concurrency, > > > max_active_runs, ...), tried switching off some DAGs completely (not > all > > > but most) etc., so far nothing helped. Right now, I am not really sure, > > > what else to try to identify a solve the issue. > > > > > > I am getting a bit desperate, so I would really appreciate any help > with > > > this. Thank you all in advance! > > > > > > Joe > > > > > >
