smhood commented on issue #36998:
URL: https://github.com/apache/airflow/issues/36998#issuecomment-1910544611
> are you seeing this issue when you run the airflow with single scheduler?
Can you share the details to reprice it?
>
> This requires triaging. Meanwhile, you can bump up the parallelism
configuration to a higher number to beat the leak. Or Restart the scheduler
after a certain number of iterations to rest these values.
Yes, this is when running on a single scheduler. We are utilizing the helm
chart and only overriding the following values in values.yaml.
```
images:
airflow:
repository: acrglobalp01.azurecr.io/data-platform/airflow-image
tag: branch-70e99939
pullPolicy: Always
labels:
azure.workload.identity/use: 'true'
airflowPodAnnotations:
azure.workload.identity/client-id: <redacted>
env:
- name: AIRFLOW__CORE__TEST_CONNECTION
value: Enabled
- name: AIRFLOW__LOGGING__REMOTE_LOGGING
value: 'True'
- name: AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID
value: wasb_airlow_logs
- name: AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER
value: wasb-airflow-logging
- name: AIRFLOW__KUBERNETES__WORKER_PODS_CREATION_BATCH_SIZE
value: '4'
- name: ENVIRONMENT
value: dev
- name: LOG_LEVEL
value: DEBUG
- name: AIRFLOW__SMTP__SMTP_MAIL_TO
value: <redacted>
- name: AIRFLOW__SMTP__SMTP_HOST
value: <redacted>
- name: AIRFLOW__SMTP__SMTP_PORT
value: '25'
- name: AIRFLOW__SMTP__SMTP_STARTTLS
value: 'False'
- name: AIRFLOW__SMTP__SMTP_SSL
value: 'False'
- name: AIRFLOW__SMTP__SMTP_MAIL_FROM
value: <redacted>
- name: AIRFLOW__SCHEDULER__USE_ROW_LEVEL_LOCKING
value: 'False'
ingress:
web:
enabled: true
pathType: Prefix
hosts:
- name: dev-airflow.privatelink.eastus2.azmk8s.io
tls:
enabled: true
secretName: dev-airflow-tls
ingressClassName: nginx
annotations:
cert-manager.io/cluster-issuer: aks-ca-cluster-issuer
workers:
serviceAccount:
annotations:
azure.workload.identity/client-id: <redacted>
resources:
requests:
cpu: '0.5'
memory: 128Mi
limits:
cpu: '1'
memory: 512Mi
executor: KubernetesExecutor
allowPodLaunching: true
webserver:
replicas: 3
podDisruptionBudget:
enabled: true
config:
maxUnavailable: 1
serviceAccount:
annotations:
azure.workload.identity/client-id: <redacted>
webserverConfigConfigMapName: webserverconfig
volumes:
- name: secrets-store-inline
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: airflow-aks-secrets
volumeMounts:
- name: secrets-store-inline
mountPath: /mnt/secrets-store
readOnly: true
dags:
persistence:
enabled: false
createUserJob:
useHelmHooks: false
applyCustomEnv: false
extraEnvFrom: |-
- secretRef:
name: airflow-azure-oauth
migrateDatabaseJob:
useHelmHooks: false
applyCustomEnv: false
jobAnnotations:
argocd.argoproj.io/hook: Sync
postgresql:
enabled: false
metadataConnection:
sslmode: require
pgbouncer:
enabled: true
data:
metadataSecretName: postgresql-connection-url
webserverSecretKeySecretName: airflow-webserver-secret-key
fernetKeySecretName: airflow-fernet-key
triggerer:
replicas: 1
serviceAccount:
annotations:
azure.workload.identity/client-id: <redacted>
args: ["bash", "-c", "exec airflow scheduler --verbose"]
config:
kubernetes_executor:
namespace: orch-dataplatform
webserver:
base_url: https://dev-airflow.privatelink.eastus2.azmk8s.io
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
