dstandish commented on a change in pull request #13571:
URL: https://github.com/apache/airflow/pull/13571#discussion_r571915182
##########
File path: chart/README.md
##########
@@ -349,6 +350,8 @@ helm install airflow . \
--set workers.persistence.enabled=false
```
+You can also configure the number of tasks that each keda worker will take
with the `config.celery.worker_concurrency` value.
Review comment:
OK I see, the important point you are making here is that you need to
set celery concurrency through helm in order to have KEDA scaling be consistent
with celery config. E.g. if you use airflow.cfg, orr env vars, they won't
necessarily agree.
I might just include the query template to make it explicit.
Here's another crack at it:
> KEDA will derive the desired number of celery workers by querying the
metastore: <the query>. You should set celery worker concurrency through helm
value `config.celery.worker_concurrency` (i.e. instead of airflow.cfg or
environment variables) so that the KEDA trigger will be consistent with worker
concurrency.
##########
File path: chart/README.md
##########
@@ -349,6 +350,8 @@ helm install airflow . \
--set workers.persistence.enabled=false
```
+You can also configure the number of tasks that each keda worker will take
with the `config.celery.worker_concurrency` value.
Review comment:
OK I see, the important point you are making here is that you need to
set celery concurrency through helm in order to have KEDA scaling be consistent
with celery config. E.g. if you use airflow.cfg, orr env vars, they won't
necessarily agree.
I might just include the query template to make it explicit.
Here's another crack at it:
> KEDA will derive the desired number of celery workers by querying the
metastore: put query here in code block. You should set celery worker
concurrency through helm value `config.celery.worker_concurrency` (i.e. instead
of airflow.cfg or environment variables) so that the KEDA trigger will be
consistent with worker concurrency.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
