Probably more of a configuration constellation issue than version specific or even an 'issue' per se. As noted, on restart the scheduler reschedules everything. I had a heavy SubDAG that when rescheduled could produce many extra tasks and a small fixed number of Celery workers. So, the scheduled tasks wouldn't be done by the time of the scheduler restart and then the scheduler would reschedule the SubDAG... debugging hilarity followed from there.
On Wed, Jun 7, 2017, 10:57 AM Jason Chen <[email protected]> wrote: > I am using Airflow 1.7.1.3 with CeleryExecutor, but not run into this > issue. > I am wondering if this issue is only for 1.8.x ? > > On Wed, Jun 7, 2017 at 8:34 AM, Russell Pierce <[email protected] > > > wrote: > > > Depending on how fast you can clear down your queue, -n can be harmful > and > > really stack up your celery queue. Keep an eye on your queue depth of you > > see a ton of messages about the task already having been run. > > > > On Mon, Jun 5, 2017, 9:18 AM Josef Samanek <[email protected]> > wrote: > > > > > Hey. Thanks for the answer. I previously also tried to run scheduler -n > > > 10, but it was back when I was still using LocalExecutor. And it did > not > > > help. I have not yet tried to do it with CeleryExecutor, so I might. > > > > > > Still, I would prefer to find an actual solution for the underlying > > > problem, not just a workaround (eventhough a working workaround is also > > > appreciated). > > > > > > Best regards, > > > Joe > > > > > > On 2017-06-02 00:10 (+0200), Alex Guziel <[email protected]. > > INVALID> > > > wrote: > > > > We've noticed this with celery, relating to this > > > > https://github.com/celery/celery/issues/3765 > > > > > > > > We also use `-n 5` option on the scheduler so it restarts every 5 > runs, > > > > which will reset all queued tasks. > > > > > > > > Best, > > > > Alex > > > > > > > > On Thu, Jun 1, 2017 at 2:18 PM, Josef Samanek < > [email protected] > > > > > > > wrote: > > > > > > > > > Hi! > > > > > > > > > > We have a problem with our airflow. Sometimes, several tasks get > > queued > > > > > but they never get run and remain in Queud state forever. Other > tasks > > > from > > > > > the same schedule interval run. And next schedule interval runs > > > normally > > > > > too. But these several tasks remain queued. > > > > > > > > > > We are using Airflow 1.8.1. Currently with CeleryExecutor and > redis, > > > but > > > > > we had the same problem with LocalExecutor as well (actually > > switching > > > to > > > > > Celery helped quite a bit, the problem now happens way less often, > > but > > > > > still it happens). We have 18 DAGs total, 13 active. Some have just > > 1-2 > > > > > tasks, but some are more complex, like 8 tasks or so and with > > > upstreams. > > > > > There are also ExternalTaskSensor tasks used. > > > > > > > > > > I tried playing around with DAG configurations (limiting > concurrency, > > > > > max_active_runs, ...), tried switching off some DAGs completely > (not > > > all > > > > > but most) etc., so far nothing helped. Right now, I am not really > > sure, > > > > > what else to try to identify a solve the issue. > > > > > > > > > > I am getting a bit desperate, so I would really appreciate any help > > > with > > > > > this. Thank you all in advance! > > > > > > > > > > Joe > > > > > > > > > > > > > > >
