kaxil commented on a change in pull request #8695: URL: https://github.com/apache/airflow/pull/8695#discussion_r422158345
########## File path: airflow/config_templates/default_airflow.cfg ########## @@ -550,6 +550,15 @@ worker_concurrency = 8 # Example: worker_autoscale = 16,12 # worker_autoscale = +# Due to an issue with Celery, worker_prefetch_multiplier must also be set to an int +# greater than 1 for autoscaling to work. Be aware that this causes the number of +# records that a worker prefetchs to be greater than the available worker processes. +# If you have multiple workers this can cause some jobs to be unnecessarily blocked if +# one worker prefetches tasks and those tasks are blocked by long running work then another +# worker with available process workers can't grab those tasks. +# https://docs.celeryproject.org/en/stable/userguide/optimizing.html#prefetch-limits +# worker_prefetch_multiplier = Review comment: Yes sorry I meant, it doesn't have to be commented out so as the default is 1 :) ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
