V0lantis commented on issue #21225:
URL: https://github.com/apache/airflow/issues/21225#issuecomment-1230793049
Hey @arkadiusz-bach,
I am sorry, that's true that I haven't posted any logs, maybe I haven't been
clear as well...
> @V0lantis task might stuck in the queued state at least due to:
>
> * Redis crashed after receiving the task and it had no time / was not
configured to save its state to disk, so the task was lost, but scheduler
thinks that task is waiting to be picked up workers
We use ElasticCache service. The service is configured to store data on disk
and is too sized for the actual amount of tasks it received and as stated
above, celery actually acknowledged that task was successfully sent to queue.
But why not? I haven't checked the log in redis, but I would think I would see
an error in celery worker.
> * terminationGracePeriodSeconds on your worker PODs is too low or it is
not there at all.(default is 60 seconds)
The warm shutdown was received 10 minutes before the pod was actually
killed. But the task was sent and acknowledged by `CeleryExecutor` with:
```
Setting external_id for <TaskInstance: example_bash_operator.this_will_skip
scheduled__2022-08-28T00:00:00+00:00 [queued]> to
979a100a-5012-4135-b01f-b93577b39ede
```
in the scheduler_log, which actually means that the task was received by
redis (or celery would through a connection error).
> This message `worker: Warm shutdown` means that celery received SIGTERM
signal and it started gracefull shutdown - it is not going to pick any more
tasks from redis queue and it will wait for all of the running tasks to finish
Yep, but as I said in my comment :
> this log message (`worker: Warm shutdown`) appeared a **couple of
milliseconds AFTER** the scheduler has sent the task to redis.
⬆️ I still think that Celery consumer exited during the handling of my
message
[here](https://github.com/celery/kombu/blob/master/kombu/transport/redis.py#L356-L370)
and didn't have time to put it in the `unacked` hash.
> But if you've got some tasks that may be running for longer than
terminationGracePeriod then Kubernetes might send SIGKILL first and:
>
> * Celery will not be able to wait for all of the running tasks to
finish(those will end with failed status)
> * it was able to pick the task from queue(before SIGTERM), but not able to
change its state to running(After SIGKILL) - maybe your case
⬆️ I stil thinks that my pods received SIGKILL 10 minutes after the SIGTERM
but I haven't been yet able to check the `terminationGracePeriod`
> Also some of the celery workers might receive SIGKILL signal, when there
is not enough memory allocated and it may led to the same behaviour,
unfortunately you may not see OOM events in the kubernetes cluster when it
happens, becaue when there is more than one process running on the container in
Kubernetes then it is chosing randomly one of the child processes within
container and sends SIGKILL(Celery is running with Main process and child
processes(workers)).
Already check as well. We have a Grafana dash to monitor the hardware
resources that the pod is consuming, and the workers were far below the
threshold we gave them
> If the Main process receives SIGKILL you will probably see OOM
event(container will restart/stop), but if child then tasks it was processing
will fail(in the logs you will be able to see that it received SIGKILL singal)
or stuck in queued state if it was able to pick it and container will be
running fine, - one of the celery processes will be restarted
The thing is, the task never started. Thanks anyway for your input, it was
really enriching :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
