nadflinn commented on issue #8480: URL: https://github.com/apache/airflow/issues/8480#issuecomment-623720953
Following up on this, it's not ideal, but if in 1.10.9 you want to set a specific worker process count to, for example, 8 workers then just include: `worker_autoscale = 8,8` in your `airflow.cfg`. As @eeshugerman pointed out, auto-scaling is not working so even if you set the upper bound to be something higher you would still sit at 8 workers. A caveat to this, I dug into auto-scaling not working and I think autoscaling is only broken in celery with the combination of settings: `worker_prefetch_multipler=1` and `task_acks_late=True` which are the settings in airflow. I submitted a [patch](https://github.com/celery/celery/pull/6069) to Celery that should fix this. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
