This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 4fa8e45c92 prevent celery command autoscale misconfig (#36576)
4fa8e45c92 is described below
commit 4fa8e45c9222f05cabef543c8fd33f737826ebe3
Author: Bowrna <[email protected]>
AuthorDate: Mon Jan 29 00:47:32 2024 +0530
prevent celery command autoscale misconfig (#36576)
---
airflow/providers/celery/provider.yaml | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/airflow/providers/celery/provider.yaml
b/airflow/providers/celery/provider.yaml
index fd63a5685a..947dfcb6b1 100644
--- a/airflow/providers/celery/provider.yaml
+++ b/airflow/providers/celery/provider.yaml
@@ -119,9 +119,10 @@ config:
default: "16"
worker_autoscale:
description: |
- The maximum and minimum concurrency that will be used when starting
workers with the
- ``airflow celery worker`` command (always keep minimum processes,
but grow
- to maximum if necessary). Note the value should be
max_concurrency,min_concurrency
+ The maximum and minimum number of pool processes that will be used
to dynamically resize
+ the pool based on load.Enable autoscaling by providing
max_concurrency,min_concurrency
+ with the ``airflow celery worker`` command (always keep minimum
processes,
+ but grow to maximum if necessary).
Pick these numbers based on resources on worker box and the nature
of the task.
If autoscale option is available, worker_concurrency will be ignored.
https://docs.celeryq.dev/en/latest/reference/celery.bin.worker.html#cmdoption-celery-worker-autoscale
