dstandish commented on code in PR #30914:
URL: https://github.com/apache/airflow/pull/30914#discussion_r1182768759
##########
chart/values.yaml:
##########
@@ -704,6 +718,51 @@ scheduler:
# Annotations to add to scheduler kubernetes service account.
annotations: {}
+
+ # Allow KEDA autoscaling.
+ keda:
+ enabled: false
+ namespaceLabels: {}
+
+ # How often KEDA polls the airflow DB to report new scale requests to the
HPA
+ pollingInterval: 5
+
+ # How many seconds KEDA will wait before scaling to zero.
+ # Note that HPA has a separate cooldown period for scale-downs
+ cooldownPeriod: 30
+
+ # Minimum number of workers created by keda
+ minReplicaCount: 0
+
+ # Maximum number of workers created by keda
+ maxReplicaCount: 10
+
+ # Specify HPA related options
+ advanced: {}
+ # horizontalPodAutoscalerConfig:
+ # behavior:
+ # scaleDown:
+ # stabilizationWindowSeconds: 300
+ # policies:
+ # - type: Percent
+ # value: 100
+ # periodSeconds: 15
+
+ # Specify the KEDA Triggers for autoscaling
+ triggers: >-
+ - type: postgresql
+ metadata:
+ targetQueryValue: "1"
+ connectionFromEnv: AIRFLOW_CONN_AIRFLOW_DB
+ query: >-
+ SELECT ceil(COUNT(*)::decimal / {{ .Values.config.core.parallelism
}})
+ FROM task_instance
+ WHERE (state='running' OR state='queued')
+ {{- if eq .Values.executor "CeleryKubernetesExecutor" }}
+ AND queue != '{{
.Values.config.celery_kubernetes_executor.kubernetes_queue }}'
+ {{- end }}
Review Comment:
i believe when task is deferred the TI immediately goes to state `deferred`.
see where TaskDeferred is caught.
then immediately a trigger record is created. initially the trigger is
unassigned. i think triggerer_id column will be null.
you'll know that the trigger has been picked up when that column,
triggerer_id, gets populated.
when triggers are done running they are deleted from db. so to get a count
of all triggers both running / assigned / otherwise, you could probably just do
`select count(*) from trigger` this would seem to be equivalent to the query
above. then you'd divide by triggerer concurrency to get something like rough
amount of triggerer instances you need. but this is just a rough sketch i
guess -- might want to think it through a bit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
