dashton90 commented on issue #13542: URL: https://github.com/apache/airflow/issues/13542#issuecomment-1146436913
Specific to MWAA, our team had a similar issue. We found this to be an issue with the MWAA default value of `celery.worker_autoscale`. The value varies by environment class, but is set to 5 * vCPU per worker. This overloads the workers, and causes resources to lock. Workers have a different number of vCPUs per environment. Defaults are: | Environment Class | Worker CPU | Default Value | | ----------------- | ---------- | ----------------------------- | | mw1.small | 1 vCPU | celery.worker_autoscale=5,5 | | mw1.medium | 2 vCPU | celery.worker_autoscale=10,10 | | mw1.large | 4 vCPU | celery.worker_autoscale=20,20 | Our resolution was to lower the value of `celery.worker_autoscale` to 2 * vCPU per worker A manageable value per environment class is | Environment Class | Parameter | | ----------------- | --------------------------- | | mw1.small | celery.worker_autoscale=2,2 | | mw1.medium | celery.worker_autoscale=4,4 | | mw1.large | celery.worker_autoscale=8,8 | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
