nadflinn commented on a change in pull request #8695:
URL: https://github.com/apache/airflow/pull/8695#discussion_r422152685
##########
File path: airflow/config_templates/config.yml
##########
@@ -1130,6 +1130,19 @@
type: string
example: 16,12
default: ~
+ - name: worker_prefetch_multiplier
+ description: |
+ Due to an issue with Celery, worker_prefetch_multiplier must also be
set to an int
+ greater than 1 for autoscaling to work. Be aware that this causes the
number of
+ records that a worker prefetchs to be greater than the available
worker processes.
+ If you have multiple workers this can cause some jobs to be
unnecessarily blocked if
+ one worker prefetches tasks and those tasks are blocked by long
running work then another
+ worker with available process workers can't grab those tasks.
+
https://docs.celeryproject.org/en/stable/userguide/optimizing.html#prefetch-limits
+ version_added: ~
+ type: int
+ example: ~
+ default: ~
Review comment:
I added an `example` instead of a `default` so that
`worker_prefetch_multiplier` was commented out after running
`pre_commit_yaml_to_cfg.py` to generate `default_airflow.cfg`. With a
`default` the setting is not commented out.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
