[
https://issues.apache.org/jira/browse/AIRFLOW-363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16442170#comment-16442170
]
Daniel Lamblin commented on AIRFLOW-363:
----------------------------------------
I'm not sure, but I think #3 was done, right [~criccomini]?
> Celery Queue becoming too huge when worker unavailable
> ------------------------------------------------------
>
> Key: AIRFLOW-363
> URL: https://issues.apache.org/jira/browse/AIRFLOW-363
> Project: Apache Airflow
> Issue Type: Bug
> Components: celery
> Affects Versions: Airflow 1.7.1.3
> Reporter: Vineet Goel
> Priority: Major
>
> The scheduler currently queues executable tasks every run of the scheduler. I
> ran into an issue where I was using a single worker and my dag was composed
> of a bunch of tasks that could run simultaneously. I was setting a
> concurrency on the worker to control how many of those tasks could run at the
> same time. However, my scheduler kept queueing more and more tasks. This can
> cause a HUGE back pressure on the celery queue, which becomes unmanageable at
> times.
> We should set the
> http://docs.celeryproject.org/en/latest/configuration.html#celery-event-queue-ttl
> configuration to handle this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
