Miretpl commented on issue #63580: URL: https://github.com/apache/airflow/issues/63580#issuecomment-5319251081
I tested the latest version of Celery 5.6.3, and the issue seems to be fixed. There were a couple of issues connected to the fix on Celery itself and Kombu. After a restart of Redis, Celery workers lose connection to it, and when it gets back, it reconnects (if it managed to do it within the configured retries). Consuming and running new tasks are not happening immediately, but it happens (on my testing setup, it was a couple of minutes after reconnecting). Currently used in the Helm Chart, `celery inspect ping` is the official way of checking if everything is fine with the worker, and I think that we should stay with it. If there is any issue with Celery, we should report it to the Celery maintainers and potentially help to fix it. Any different kind of handling of it may cause unnecessary and disruptive restarts for the Airflow instances, which I think we don't want to. I will leave this issue open for anyone to jump on it potentially. If there are no other comments till the end of the week, I will close it as not planned. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
