dstandish commented on code in PR #30914:
URL: https://github.com/apache/airflow/pull/30914#discussion_r1185744542
##########
chart/values.yaml:
##########
@@ -704,6 +718,51 @@ scheduler:
# Annotations to add to scheduler kubernetes service account.
annotations: {}
+
+ # Allow KEDA autoscaling.
+ keda:
+ enabled: false
+ namespaceLabels: {}
+
+ # How often KEDA polls the airflow DB to report new scale requests to the
HPA
+ pollingInterval: 5
+
+ # How many seconds KEDA will wait before scaling to zero.
+ # Note that HPA has a separate cooldown period for scale-downs
+ cooldownPeriod: 30
+
+ # Minimum number of workers created by keda
+ minReplicaCount: 0
+
+ # Maximum number of workers created by keda
+ maxReplicaCount: 10
+
+ # Specify HPA related options
+ advanced: {}
+ # horizontalPodAutoscalerConfig:
+ # behavior:
+ # scaleDown:
+ # stabilizationWindowSeconds: 300
+ # policies:
+ # - type: Percent
+ # value: 100
+ # periodSeconds: 15
+
+ # Specify the KEDA Triggers for autoscaling
+ triggers: >-
+ - type: postgresql
+ metadata:
+ targetQueryValue: "1"
+ connectionFromEnv: AIRFLOW_CONN_AIRFLOW_DB
+ query: >-
+ SELECT ceil(COUNT(*)::decimal / {{ .Values.config.core.parallelism
}})
+ FROM task_instance
+ WHERE (state='running' OR state='queued')
+ {{- if eq .Values.executor "CeleryKubernetesExecutor" }}
+ AND queue != '{{
.Values.config.celery_kubernetes_executor.kubernetes_queue }}'
+ {{- end }}
Review Comment:
@asosnovsky i don't think so. i don't think the query needs to be perfect.
but at least it shouldn't be completely non-sensical, which is the case right
now. the query should have some basic justifiability even if it's not perfect.
don't you think? otherwise users will naively use this feature thinking it
has a sensible default and find that the scaling has literally nothing to do
with how many triggers are running.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
