enanablancaynumeros removed a comment on issue #15799:
URL: https://github.com/apache/airflow/issues/15799#issuecomment-869594168
> Note that Celery 5 is not YET officially supported by Airflow. There is an
open issue for that #11301 - and if you would like to make a PR to add support
for it, you are most welcome!
I'm not using Airflow although I'm running long running task with late ack,
the rest of the options are:
```
worker_prefetch_multiplier = 1
worker_concurrency = int(os.environ.get("WORKER_CONCURRENCY", 1))
worker_max_tasks_per_child = 1
worker_send_task_events = True
worker_redirect_stdouts = False
worker_cancel_long_running_tasks_on_connection_loss = True
task_send_sent_event = True
task_ignore_result = False
task_serializer = "json"
task_acks_late = strtobool(os.environ.get("CELERY_ACKS_LATE", "True"))
task_always_eager = strtobool(os.environ.get("CELERY_EAGER", "False"))
task_eager_propagates = True # if eager == True the this applies
task_track_started = True
task_time_limit = 86400 * 2 # 48 hours in seconds
```
I'm requesting to cloudamqp a cluster downgrade in the meantime to 3.8.14
while we can clarify a solution from the application side and/or an increase in
the consumer_timeout parameter.
**EDIT**
The behavior we are experiencing is that once a process raises this
exception, the task seems to be rerouted to another consumer but the processes
keep running in some sort of zombie mode. It looks that is also duplicating
this way the tasks, despite now having
`worker_cancel_long_running_tasks_on_connection_loss = True`.
I can't offer anything more useful at the moment!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
