turbaszek edited a comment on issue #8480: URL: https://github.com/apache/airflow/issues/8480#issuecomment-617194692
Duplicate of https://github.com/apache/airflow/issues/8044 @jsmodic you are right, autoscaling is weird in Celery in 4.x and there are issues reporting this problem in Celery itself. If you want to perform production-grade autoscaling and you are using k8s cluster this may be interesting to you: https://www.astronomer.io/blog/the-keda-autoscaler/ I tested it on master and it seems to be working fine (`-c n` spawns main celery process and `n` child processes). ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
