dstandish commented on a change in pull request #13571:
URL: https://github.com/apache/airflow/pull/13571#discussion_r570734550
##########
File path: chart/templates/workers/worker-kedaautoscaler.yaml
##########
@@ -36,13 +36,13 @@ metadata:
spec:
scaleTargetRef:
deploymentName: {{ .Release.Name }}-worker
- pollingInterval: {{ .Values.workers.keda.pollingInterval }} # Optional.
Default: 30 seconds
- cooldownPeriod: {{ .Values.workers.keda.cooldownPeriod }} # Optional.
Default: 300 seconds
- maxReplicaCount: {{ .Values.workers.keda.maxReplicaCount }} # Optional.
Default: 100
+ pollingInterval: {{ .Values.workers.keda.pollingInterval }} # Optional.
Default: 30 seconds
+ cooldownPeriod: {{ .Values.workers.keda.cooldownPeriod }} # Optional.
Default: 300 seconds
+ maxReplicaCount: {{ .Values.workers.keda.maxReplicaCount }} # Optional.
Default: 100
triggers:
- type: postgresql
metadata:
targetQueryValue: "1"
connection: AIRFLOW_CONN_AIRFLOW_DB
- query: "SELECT ceil(COUNT(*)::decimal / 16) FROM task_instance WHERE
state='running' OR state='queued'"
+ query: "SELECT ceil(COUNT(*)::decimal / {{
.Values.config.celery.worker_concurrency }}) FROM task_instance WHERE
state='running' OR state='queued'"
Review comment:
i wonder if a better solution would be to allow user to specify the
entire query, instead of `worker_concurrency`. ~for one, this would resolve
the problem highlighted in my other comment~. additionally though it would
open the door to support for mysql. mysql has a different query syntax but
otherwise there is a keda trigger for it and we could add support for it by
enabling custom query. custom query is also more flexible in general.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
