asosnovsky commented on code in PR #30914:
URL: https://github.com/apache/airflow/pull/30914#discussion_r1185666839
##########
chart/values.yaml:
##########
@@ -704,6 +718,51 @@ scheduler:
# Annotations to add to scheduler kubernetes service account.
annotations: {}
+
+ # Allow KEDA autoscaling.
+ keda:
+ enabled: false
+ namespaceLabels: {}
+
+ # How often KEDA polls the airflow DB to report new scale requests to the
HPA
+ pollingInterval: 5
+
+ # How many seconds KEDA will wait before scaling to zero.
+ # Note that HPA has a separate cooldown period for scale-downs
+ cooldownPeriod: 30
+
+ # Minimum number of workers created by keda
+ minReplicaCount: 0
+
+ # Maximum number of workers created by keda
+ maxReplicaCount: 10
+
+ # Specify HPA related options
+ advanced: {}
+ # horizontalPodAutoscalerConfig:
+ # behavior:
+ # scaleDown:
+ # stabilizationWindowSeconds: 300
+ # policies:
+ # - type: Percent
+ # value: 100
+ # periodSeconds: 15
+
+ # Specify the KEDA Triggers for autoscaling
+ triggers: >-
+ - type: postgresql
+ metadata:
+ targetQueryValue: "1"
+ connectionFromEnv: AIRFLOW_CONN_AIRFLOW_DB
+ query: >-
+ SELECT ceil(COUNT(*)::decimal / {{ .Values.config.core.parallelism
}})
+ FROM task_instance
+ WHERE (state='running' OR state='queued')
+ {{- if eq .Values.executor "CeleryKubernetesExecutor" }}
+ AND queue != '{{
.Values.config.celery_kubernetes_executor.kubernetes_queue }}'
+ {{- end }}
Review Comment:
@dstandish perhaps this topic should be on a separate conversation?
our company atm is using a semi-hacked way of this (we basically set
replicas to `~` and then manually defined the same definitions I've added
here). I will add some test cases over the weekend (or monday) and after that
it would be nice to merge this :)
In the meantime maybe we should transfer this to a github issue on how to
improve the queries? I feel like it should be ok, since my change allows people
to set whatever query they want.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
