dstandish commented on a change in pull request #13571:
URL: https://github.com/apache/airflow/pull/13571#discussion_r570733754
##########
File path: chart/values.yaml
##########
@@ -709,6 +709,7 @@ config:
rbac: 'True'
celery:
default_queue: celery
+ worker_concurrency: 16
Review comment:
this seems potentially misleading because, if i understand correctly, it
affects only the keda query but not the actual work concurrency setting.
##########
File path: chart/templates/workers/worker-kedaautoscaler.yaml
##########
@@ -36,13 +36,13 @@ metadata:
spec:
scaleTargetRef:
deploymentName: {{ .Release.Name }}-worker
- pollingInterval: {{ .Values.workers.keda.pollingInterval }} # Optional.
Default: 30 seconds
- cooldownPeriod: {{ .Values.workers.keda.cooldownPeriod }} # Optional.
Default: 300 seconds
- maxReplicaCount: {{ .Values.workers.keda.maxReplicaCount }} # Optional.
Default: 100
+ pollingInterval: {{ .Values.workers.keda.pollingInterval }} # Optional.
Default: 30 seconds
+ cooldownPeriod: {{ .Values.workers.keda.cooldownPeriod }} # Optional.
Default: 300 seconds
+ maxReplicaCount: {{ .Values.workers.keda.maxReplicaCount }} # Optional.
Default: 100
triggers:
- type: postgresql
metadata:
targetQueryValue: "1"
connection: AIRFLOW_CONN_AIRFLOW_DB
- query: "SELECT ceil(COUNT(*)::decimal / 16) FROM task_instance WHERE
state='running' OR state='queued'"
+ query: "SELECT ceil(COUNT(*)::decimal / {{
.Values.config.celery.worker_concurrency }}) FROM task_instance WHERE
state='running' OR state='queued'"
Review comment:
i wonder if a better solution would be to allow user to specify the
entire query, instead of `worker_concurrency`. for one, this would resolve the
problem highlighted in my other comment. additionally though it would open the
door to support for mysql. mysql has a different query syntax but otherwise
there is a keda trigger for it and we could add support for it by enabling
custom query. custom query is also more flexible in general.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
