dimon222 commented on issue #41163: URL: https://github.com/apache/airflow/issues/41163#issuecomment-2282236848
Regarding first proposed solution - HPA waiting for completion of work: I'm afraid this is not fixable by Airflow project, as it's the limitation of Kubernetes that during scale down operation request you can't choose the "wait for completion of work. The timing for graceful shutdown has to be configured on reason at start time, you can't update this when the container of celery worker is already running (the resource spec is immutable, you have to trigger "deploy" style operation to bounce container to do any changes). I have been waiting for this for years, but K8s development community isn't working on it currently. https://github.com/kubernetes/enhancements/issues/2255#issuecomment-2261281830 Not sure about second proposed solution though. I'm not certain is Kubernetes sends upfront signal to remind that shutdown signal will come in grace period amount of seconds. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
