jsmodic edited a comment on issue #8480:
URL: https://github.com/apache/airflow/issues/8480#issuecomment-617282541
@turbaszek what celery version are you using? On 1.10.9 it starts up like
this for me on Celery 4.3.0 with no autoscaling configured anywhere.
From Celery's stdout on startup:
> [config]
app: airflow.executors.celery_executor:0x7fe57b350be0
transport: {my_transport}
results: {my_db}
concurrency: {min=12, max=16} (prefork)
task events: OFF (enable -E to monitor tasks in this worker)
On the machine:
> ps aux | grep celery
[celeryd: celery@ip-my-ip:MainProcess] -active- (worker -q queue -c 4)
[celeryd: celery@ip-my-ip:ForkPoolWorker-1]
[celeryd: celery@ip-my-ip:ForkPoolWorker-2]
[celeryd: celery@ip-my-ip:ForkPoolWorker-3]
[celeryd: celery@ip-my-ip:ForkPoolWorker-4]
[celeryd: celery@ip-my-ip:ForkPoolWorker-5]
[celeryd: celery@ip-my-ip:ForkPoolWorker-6]
[celeryd: celery@ip-my-ip:ForkPoolWorker-7]
[celeryd: celery@ip-my-ip:ForkPoolWorker-8]
[celeryd: celery@ip-my-ip:ForkPoolWorker-9]
[celeryd: celery@ip-my-ip:ForkPoolWorker-10]
[celeryd: celery@ip-my-ip:ForkPoolWorker-11]
[celeryd: celery@ip-my-ip:ForkPoolWorker-12]
There's only 4 cores on that machine, that's what triggered me digging into
how there could possibly be 12 workers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
