jsmodic opened a new issue #8480: URL: https://github.com/apache/airflow/issues/8480
Airflow 1.10.9 Celery 4.3.0 Setting Celery 'worker_concurrency' is overriden by the autoscaling no matter what you configure. Problematic code is here: https://github.com/apache/airflow/blob/6db66eaf927a50f2dcd85d0ae6033f7ea3699099/airflow/cli/commands/celery_command.py#L121 (this is head but it's the same in 1.10.x) You always get the default autoscaling setting with head if you don't set autoscaling at all but do set worker concurrency (which is 12 workers). Removing "autoscale" from the dictionary when autoscaling is not set is enough to solve this (at least, I tested the worker_concurrency half of things, and the asked for concurrency is back with that change). ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
