This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new e161af12209 Stop shipping metadata DB credentials to task-executing
components (#72206)
e161af12209 is described below
commit e161af1220976ea148dc019a456124721b659c69
Author: Jarek Potiuk <[email protected]>
AuthorDate: Sat Nov 14 01:45:08 2026 +0100
Stop shipping metadata DB credentials to task-executing components (#72206)
* Stop shipping metadata DB credentials to task-executing components
standard_airflow_environment emitted AIRFLOW__DATABASE__SQL_ALCHEMY_CONN and
AIRFLOW_CONN_AIRFLOW_DB for every component that included it, which put the
metadata database connection string into Celery worker pods and into every
KubernetesExecutor task pod. Task code inherits its container's
environment, so
Dag-author code could reach the database directly.
Callers now state whether the component uses the database, through an
IncludeMetadataDb flag on the include - the same shape as the existing
IncludeJwtSecret. Components that talk to the database pass true; task pods
pass
false. In Airflow 3 the task SDK reaches the Execution API, so a task pod
has no
functional need for the connection.
Celery workers are the one conditional case. When KEDA autoscales them its
ScaledObject reads the connection from an env var on the worker pod spec,
so the
variable has to stay for KEDA to find it; workers therefore pass the
existing
$keda variable rather than a constant. KEDA is off by default, so a default
deployment no longer carries the credentials, and deployments that enable
it can
point KEDA at a restricted connection.
Adds regression tests on both surfaces and gives the worker its own
expected env
list in the two tests that assert the full set.
* Keep the metadata DB on the worker's migration-wait init container
That init container queries the migration state, so denying it the database
left it unable to connect and the worker pod never left Init - no Celery
task
could run at all. It exits before the worker starts and shares no
environment
with the container that executes task code, so the credentials there are not
reachable from a Dag.
* Extract the metadata DB credentials into their own chart helper
Review feedback: threading an `IncludeMetadataDb` flag through
`standard_airflow_environment` put per-component policy inside a helper
shared by every component. A dedicated include, shaped like the existing
`jwt_secret_environment`, lets each template state for itself whether it
needs the credentials, and drops the twenty `merge (dict ...)` call sites.
Generated-by: Claude Code (Opus 5)
Claude-Session: https://claude.ai/code/session_01XS3bodTDYYGrPmorhtLsjP
* Apply review suggestions to the chart DB-credential changes
Shorter template comments that do not tie the helper to the current
execution model, one KEDA conditional instead of two, and tighter test names
and assertions.
Generated-by: Claude Opus 5
* Keep Celery results working when workers lose the metadata DB
Celery derives its result backend from sql_alchemy_conn when none is set,
so a worker without it wrote results to a pod-local SQLite file the scheduler
never reads, and tasks never finished. The airflow_db connection on the API
server also let tasks fetch the same credentials through the Execution API.
Generated-by: Claude Opus 5
---
chart/docs/production-guide.rst | 18 ++
chart/files/pod-template-file.kubernetes-helm-yaml | 4 +-
chart/newsfragments/72206.significant.rst | 3 +
chart/templates/_helpers.yaml | 80 +++++++--
.../api-server/api-server-deployment.yaml | 2 +
chart/templates/cleanup/cleanup-cronjob.yaml | 1 +
.../dag-processor/dag-processor-deployment.yaml | 2 +
.../database-cleanup/database-cleanup-cronjob.yaml | 1 +
chart/templates/flower/flower-deployment.yaml | 1 +
chart/templates/jobs/create-user-job.yaml | 1 +
chart/templates/jobs/migrate-database-job.yaml | 1 +
.../templates/scheduler/scheduler-deployment.yaml | 2 +
.../templates/triggerer/triggerer-deployment.yaml | 2 +
chart/templates/workers/worker-deployment.yaml | 8 +
.../helm_tests/airflow_aux/test_airflow_common.py | 191 ++++++++++++++++++++-
.../airflow_aux/test_pod_template_file.py | 76 ++++++++
16 files changed, 371 insertions(+), 22 deletions(-)
diff --git a/chart/docs/production-guide.rst b/chart/docs/production-guide.rst
index 137bb06a886..3be63bdb6d7 100644
--- a/chart/docs/production-guide.rst
+++ b/chart/docs/production-guide.rst
@@ -103,6 +103,24 @@ Similarly, ``resultBackendSecretKey`` and
``brokerUrlSecretKey`` configure the k
used with ``resultBackendSecretName`` and ``brokerUrlSecretName``. All of
these default to
``connection`` and only apply when the corresponding secret name is provided.
+Keeping database credentials away from Celery workers
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Task code runs on Celery workers, so anything in the worker's environment is
readable by it. Celery
+workers need the metadata database only as the default Celery result backend,
and they keep
+``AIRFLOW__DATABASE__SQL_ALCHEMY_CONN`` for that reason. To remove it, give
Celery a separate result
+backend with ``data.resultBackendConnection``,
``data.resultBackendSecretName`` or
+``config.celery.result_backend``. A backend in the same database needs its own
database user, limited to the
+Celery result tables, or the workers still hold the metadata database
credentials. A backend set any other
+way, such as ``env`` or ``secret``, is not detected and the worker keeps the
connection. A separate result
+backend needs ``apache-airflow-providers-celery`` 3.12.4 or newer. KEDA reads
the
+connection from the worker's environment, so workers scaled by it always keep
the credentials.
+
+Containers without the connection fall back to SQLite, and engine arguments
that SQLite does not accept make
+Airflow fail at import. The chart clears ``sql_alchemy_engine_args`` there
when it is set through
+``config.database``, so set it that way rather than through ``env``: Kerberos
sidecars have no environment of
+their own to override it in.
+
.. _production-guide:pgbouncer:
Metadata DB Cleanup
diff --git a/chart/files/pod-template-file.kubernetes-helm-yaml
b/chart/files/pod-template-file.kubernetes-helm-yaml
index 4121fee94e2..fecd43a0dcd 100644
--- a/chart/files/pod-template-file.kubernetes-helm-yaml
+++ b/chart/files/pod-template-file.kubernetes-helm-yaml
@@ -108,6 +108,7 @@ spec:
value: {{ include "kerberos_ccache_path" . | quote }}
{{- include "custom_airflow_environment" . | indent 6 }}
{{- include "standard_airflow_environment" . | indent 6 }}
+ {{- include "kerberos_sidecar_metadata_db_environment" . | indent 6 }}
{{- end }}
containers:
- envFrom: {{- include "custom_airflow_environment_from" . | default "\n
[]" | indent 6 }}
@@ -130,7 +131,8 @@ spec:
- name: KRB5CCNAME
value: {{ include "kerberos_ccache_path" . | quote }}
{{- end }}
- {{- include "standard_airflow_environment" . | indent 6}}
+ {{- include "standard_airflow_environment" . | indent 6 }}
+ {{- include "no_metadata_db_environment" . | indent 6 }}
{{- include "custom_airflow_environment" . | indent 6 }}
{{- include "container_extra_envs" (list .
.Values.workers.kubernetes.env) | indent 6 }}
image: {{ template "pod_template_image" . }}
diff --git a/chart/newsfragments/72206.significant.rst
b/chart/newsfragments/72206.significant.rst
new file mode 100644
index 00000000000..2b5a713ccf2
--- /dev/null
+++ b/chart/newsfragments/72206.significant.rst
@@ -0,0 +1,3 @@
+KubernetesExecutor task pods no longer receive the metadata database
credentials, and Celery workers drop them when a separate result backend is
configured.
+
+``AIRFLOW__DATABASE__SQL_ALCHEMY_CONN`` and ``AIRFLOW_CONN_AIRFLOW_DB`` are
removed from KubernetesExecutor task pods and, when a result backend is set
with ``data.resultBackendConnection``, ``data.resultBackendSecretName`` or
``config.celery.result_backend``, the Celery worker container. Without such a
result backend the Celery worker keeps ``AIRFLOW__DATABASE__SQL_ALCHEMY_CONN``,
because Celery derives its result backend from it. With KEDA the worker keeps
both variables. Worker Kerber [...]
diff --git a/chart/templates/_helpers.yaml b/chart/templates/_helpers.yaml
index 9a5238fdc8c..1f0074b3c66 100644
--- a/chart/templates/_helpers.yaml
+++ b/chart/templates/_helpers.yaml
@@ -70,20 +70,6 @@ If release name contains chart name it will be used as a
full name.
name: {{ template "fernet_key_secret" . }}
key: fernet-key
{{- end }}
- {{- if
.Values.enableBuiltInSecretEnvVars.AIRFLOW__DATABASE__SQL_ALCHEMY_CONN }}
- - name: AIRFLOW__DATABASE__SQL_ALCHEMY_CONN
- valueFrom:
- secretKeyRef:
- name: {{ template "airflow_metadata_secret" . }}
- key: {{ template "airflow_metadata_secret_key" . }}
- {{- end }}
- {{- if .Values.enableBuiltInSecretEnvVars.AIRFLOW_CONN_AIRFLOW_DB }}
- - name: AIRFLOW_CONN_AIRFLOW_DB
- valueFrom:
- secretKeyRef:
- name: {{ template "airflow_metadata_secret" . }}
- key: {{ template "airflow_metadata_secret_key" . }}
- {{- end }}
{{- if .Values.enableBuiltInSecretEnvVars.AIRFLOW__API__SECRET_KEY }}
- name: AIRFLOW__API__SECRET_KEY
valueFrom:
@@ -152,6 +138,72 @@ If release name contains chart name it will be used as a
full name.
{{- end }}
{{- end }}
+{{/* Metadata database connection string, for components that talk to the
metadata DB. */}}
+{{- define "metadata_db_sql_alchemy_environment" }}
+ {{- if
.Values.enableBuiltInSecretEnvVars.AIRFLOW__DATABASE__SQL_ALCHEMY_CONN }}
+ - name: AIRFLOW__DATABASE__SQL_ALCHEMY_CONN
+ valueFrom:
+ secretKeyRef:
+ name: {{ template "airflow_metadata_secret" . }}
+ key: {{ template "airflow_metadata_secret_key" . }}
+ {{- end }}
+{{- end }}
+
+{{/*
+Metadata database credentials, including the `airflow_db` connection. The API
server must not get
+that connection: it answers the Execution API's connection lookups, so tasks
could read it from there.
+*/}}
+{{- define "metadata_db_environment" }}
+ {{- include "metadata_db_sql_alchemy_environment" . }}
+ {{- if .Values.enableBuiltInSecretEnvVars.AIRFLOW_CONN_AIRFLOW_DB }}
+ - name: AIRFLOW_CONN_AIRFLOW_DB
+ valueFrom:
+ secretKeyRef:
+ name: {{ template "airflow_metadata_secret" . }}
+ key: {{ template "airflow_metadata_secret_key" . }}
+ {{- end }}
+{{- end }}
+
+{{/*
+For containers without `sql_alchemy_conn`: Airflow falls back to an empty
SQLite database there. CLI
+commands that check migrations exit on it, and creating the engine at import
fails on engine args meant
+for the real database.
+*/}}
+{{- define "no_metadata_db_environment" }}
+ - name: AIRFLOW__DATABASE__CHECK_MIGRATIONS
+ value: "False"
+ {{- $databaseConfig := .Values.config.database | default dict }}
+ {{- if or $databaseConfig.sql_alchemy_engine_args
$databaseConfig.sql_alchemy_engine_args_cmd
$databaseConfig.sql_alchemy_engine_args_secret }}
+ - name: AIRFLOW__DATABASE__SQL_ALCHEMY_ENGINE_ARGS
+ value: "{}"
+ {{- end }}
+{{- end }}
+
+{{/*
+Before Airflow 3.4.0 `airflow kerberos` opens the metadata DB at startup, so
the sidecar keeps the
+connection there. It does not run task code, and task containers cannot read
its environment.
+*/}}
+{{- define "kerberos_sidecar_metadata_db_environment" }}
+ {{- if semverCompare "<3.4.0" .Values.airflowVersion }}
+ {{- include "metadata_db_sql_alchemy_environment" . }}
+ {{- else }}
+ {{- include "no_metadata_db_environment" . }}
+ {{- end }}
+{{- end }}
+
+{{/*
+Non-empty when Celery workers get a result backend other than the
`sql_alchemy_conn` fallback. Without
+one, a worker with no `sql_alchemy_conn` writes results to a local SQLite file
the scheduler never reads.
+A backend set in a way the chart cannot see counts as unset, which only leaves
the connection in place.
+*/}}
+{{- define "celery_result_backend_configured" -}}
+ {{- $celeryConfig := .Values.config.celery | default dict }}
+ {{- $fromSecret := and
.Values.enableBuiltInSecretEnvVars.AIRFLOW__CELERY__RESULT_BACKEND (or
.Values.data.resultBackendSecretName .Values.data.resultBackendConnection) }}
+ {{- if or $fromSecret $celeryConfig.result_backend
$celeryConfig.result_backend_cmd $celeryConfig.result_backend_secret -}}
+ true
+ {{- end }}
+{{- end }}
+
{{/* KEDA scaler database connection for the worker and triggerer autoscalers
*/}}
{{- define "keda_airflow_environment" }}
{{- if or (eq .Values.data.metadataConnection.protocol "mysql") (and
.Values.pgbouncer.enabled (not .UsePgbouncer)) }}
diff --git a/chart/templates/api-server/api-server-deployment.yaml
b/chart/templates/api-server/api-server-deployment.yaml
index de5044fdcfd..78c652c352a 100644
--- a/chart/templates/api-server/api-server-deployment.yaml
+++ b/chart/templates/api-server/api-server-deployment.yaml
@@ -156,6 +156,7 @@ spec:
env:
{{- include "custom_airflow_environment" . | indent 10 }}
{{- include "standard_airflow_environment" . | indent 10 }}
+ {{- include "metadata_db_sql_alchemy_environment" . | indent 10 }}
{{- if .Values.apiServer.waitForMigrations.env }}
{{- tpl (toYaml .Values.apiServer.waitForMigrations.env) $ |
nindent 12 }}
{{- end }}
@@ -232,6 +233,7 @@ spec:
env:
{{- include "custom_airflow_environment" . | indent 10 }}
{{- include "standard_airflow_environment" . | indent 10 }}
+ {{- include "metadata_db_sql_alchemy_environment" . | indent 10 }}
{{- include "jwt_secret_environment" . | indent 10 }}
{{- include "container_extra_envs" (list . .Values.apiServer.env)
| indent 10 }}
{{- if .Values.apiServer.extraContainers }}
diff --git a/chart/templates/cleanup/cleanup-cronjob.yaml
b/chart/templates/cleanup/cleanup-cronjob.yaml
index 556afaac6b5..23c9bd81cda 100644
--- a/chart/templates/cleanup/cleanup-cronjob.yaml
+++ b/chart/templates/cleanup/cleanup-cronjob.yaml
@@ -109,6 +109,7 @@ spec:
{{- end }}
env:
{{- include "standard_airflow_environment" . | indent 12 }}
+ {{- include "metadata_db_environment" . | indent 12 }}
{{- include "container_extra_envs" (list .
.Values.cleanup.env) | indent 12 }}
volumeMounts:
{{- include "airflow_config_mount" . | nindent 16 }}
diff --git a/chart/templates/dag-processor/dag-processor-deployment.yaml
b/chart/templates/dag-processor/dag-processor-deployment.yaml
index ca8453c2bfe..a74c0a4ca9a 100644
--- a/chart/templates/dag-processor/dag-processor-deployment.yaml
+++ b/chart/templates/dag-processor/dag-processor-deployment.yaml
@@ -141,6 +141,7 @@ spec:
env:
{{- include "custom_airflow_environment" . | indent 10 }}
{{- include "standard_airflow_environment" . | indent 10 }}
+ {{- include "metadata_db_environment" . | indent 10 }}
{{- if .Values.dagProcessor.waitForMigrations.env }}
{{- tpl (toYaml .Values.dagProcessor.waitForMigrations.env) $ |
nindent 12 }}
{{- end }}
@@ -188,6 +189,7 @@ spec:
env:
{{- include "custom_airflow_environment" . | indent 10 }}
{{- include "standard_airflow_environment" . | indent 10 }}
+ {{- include "metadata_db_environment" . | indent 10 }}
{{- include "container_extra_envs" (list .
.Values.dagProcessor.env) | indent 10 }}
livenessProbe:
initialDelaySeconds: {{
.Values.dagProcessor.livenessProbe.initialDelaySeconds }}
diff --git a/chart/templates/database-cleanup/database-cleanup-cronjob.yaml
b/chart/templates/database-cleanup/database-cleanup-cronjob.yaml
index 041425920fc..a567aad7b47 100644
--- a/chart/templates/database-cleanup/database-cleanup-cronjob.yaml
+++ b/chart/templates/database-cleanup/database-cleanup-cronjob.yaml
@@ -117,6 +117,7 @@ spec:
env:
{{- end }}
{{- include "standard_airflow_environment" . | indent 14 }}
+ {{- include "metadata_db_environment" . | indent 14 }}
{{- include "container_extra_envs" (list .
.Values.databaseCleanup.env) | indent 14 }}
volumeMounts:
{{- include "airflow_config_mount" . | nindent 16 }}
diff --git a/chart/templates/flower/flower-deployment.yaml
b/chart/templates/flower/flower-deployment.yaml
index 1fb6031c080..e6cc6e597e4 100644
--- a/chart/templates/flower/flower-deployment.yaml
+++ b/chart/templates/flower/flower-deployment.yaml
@@ -166,6 +166,7 @@ spec:
key: basicAuth
{{- end }}
{{- include "standard_airflow_environment" . | indent 10 }}
+ {{- include "metadata_db_environment" . | indent 10 }}
{{- include "custom_airflow_environment" . | indent 10 }}
{{- include "container_extra_envs" (list . .Values.flower.env) |
indent 10 }}
{{- if .Values.flower.extraContainers }}
diff --git a/chart/templates/jobs/create-user-job.yaml
b/chart/templates/jobs/create-user-job.yaml
index 2136bdc08f0..58a431ededf 100644
--- a/chart/templates/jobs/create-user-job.yaml
+++ b/chart/templates/jobs/create-user-job.yaml
@@ -117,6 +117,7 @@ spec:
env:
{{- end }}
{{- include "standard_airflow_environment" . | indent 10 }}
+ {{- include "metadata_db_environment" . | indent 10 }}
{{- include "container_extra_envs" (list .
.Values.createUserJob.env) | indent 10 }}
resources: {{- toYaml .Values.createUserJob.resources | nindent 12 }}
volumeMounts:
diff --git a/chart/templates/jobs/migrate-database-job.yaml
b/chart/templates/jobs/migrate-database-job.yaml
index 77fbb4b6fec..3370272273f 100644
--- a/chart/templates/jobs/migrate-database-job.yaml
+++ b/chart/templates/jobs/migrate-database-job.yaml
@@ -119,6 +119,7 @@ spec:
- name: PYTHONUNBUFFERED
value: "1"
{{- include "standard_airflow_environment" . | indent 10 }}
+ {{- include "metadata_db_environment" . | indent 10 }}
{{- if .Values.migrateDatabaseJob.env }}
{{- tpl (toYaml .Values.migrateDatabaseJob.env) $ | nindent 12 }}
{{- end }}
diff --git a/chart/templates/scheduler/scheduler-deployment.yaml
b/chart/templates/scheduler/scheduler-deployment.yaml
index 22c48c40bf8..31db29b80fe 100644
--- a/chart/templates/scheduler/scheduler-deployment.yaml
+++ b/chart/templates/scheduler/scheduler-deployment.yaml
@@ -169,6 +169,7 @@ spec:
env:
{{- include "custom_airflow_environment" . | indent 10 }}
{{- include "standard_airflow_environment" . | indent 10 }}
+ {{- include "metadata_db_environment" . | indent 10 }}
{{- if .Values.scheduler.waitForMigrations.env }}
{{- tpl (toYaml .Values.scheduler.waitForMigrations.env) $ |
nindent 12 }}
{{- end }}
@@ -199,6 +200,7 @@ spec:
env:
{{- include "custom_airflow_environment" . | indent 10 }}
{{- include "standard_airflow_environment" . | indent 10 }}
+ {{- include "metadata_db_environment" . | indent 10 }}
{{- include "jwt_secret_environment" . | indent 10 }}
{{- include "container_extra_envs" (list . .Values.scheduler.env)
| indent 10 }}
livenessProbe:
diff --git a/chart/templates/triggerer/triggerer-deployment.yaml
b/chart/templates/triggerer/triggerer-deployment.yaml
index 2f66a880c29..a0a4266999c 100644
--- a/chart/templates/triggerer/triggerer-deployment.yaml
+++ b/chart/templates/triggerer/triggerer-deployment.yaml
@@ -160,6 +160,7 @@ spec:
env:
{{- include "custom_airflow_environment" . | indent 10 }}
{{- include "standard_airflow_environment" . | indent 10 }}
+ {{- include "metadata_db_environment" . | indent 10 }}
{{- if .Values.triggerer.waitForMigrations.env }}
{{- tpl (toYaml .Values.triggerer.waitForMigrations.env) $ |
nindent 12 }}
{{- end }}
@@ -207,6 +208,7 @@ spec:
env:
{{- include "custom_airflow_environment" . | indent 10 }}
{{- include "standard_airflow_environment" . | indent 10 }}
+ {{- include "metadata_db_environment" . | indent 10 }}
{{- if $keda }}
{{- include "keda_airflow_environment" (merge (dict "UsePgbouncer"
.Values.triggerer.keda.usePgbouncer) .) | indent 10 }}
{{- end }}
diff --git a/chart/templates/workers/worker-deployment.yaml
b/chart/templates/workers/worker-deployment.yaml
index 86c5bef18fd..aae67b42034 100644
--- a/chart/templates/workers/worker-deployment.yaml
+++ b/chart/templates/workers/worker-deployment.yaml
@@ -236,6 +236,7 @@ spec:
value: {{ include "kerberos_ccache_path" . | quote }}
{{- include "custom_airflow_environment" . | indent 10 }}
{{- include "standard_airflow_environment" . | indent 10 }}
+ {{- include "kerberos_sidecar_metadata_db_environment" . | indent
10 }}
{{- end }}
{{- if .Values.workers.celery.waitForMigrations.enabled }}
- name: wait-for-airflow-migrations
@@ -262,7 +263,9 @@ spec:
envFrom: {{- include "custom_airflow_environment_from" . | default
"\n []" | indent 10 }}
env:
{{- include "custom_airflow_environment" . | indent 10 }}
+ {{- /* This init container queries the migration state, so it
needs the database regardless of KEDA. */}}
{{- include "standard_airflow_environment" . | indent 10 }}
+ {{- include "metadata_db_environment" . | indent 10 }}
{{- if .Values.workers.celery.waitForMigrations.env }}
{{- tpl (toYaml .Values.workers.celery.waitForMigrations.env) $
| nindent 12 }}
{{- end }}
@@ -349,7 +352,12 @@ spec:
{{- include "custom_airflow_environment" . | indent 10 }}
{{- include "standard_airflow_environment" . | indent 10 }}
{{- if $keda }}
+ {{- include "metadata_db_environment" . | indent 10 }}
{{- include "keda_airflow_environment" (merge (dict "UsePgbouncer"
.Values.workers.celery.keda.usePgbouncer) .) | indent 10 }}
+ {{- else if not (include "celery_result_backend_configured" .) }}
+ {{- include "metadata_db_sql_alchemy_environment" . | indent 10 }}
+ {{- else }}
+ {{- include "no_metadata_db_environment" . | indent 10 }}
{{- end }}
{{- include "container_extra_envs" (list .
.Values.workers.celery.env) | indent 10 }}
{{- if .Values.workers.celery.kerberosSidecar.enabled }}
diff --git a/chart/tests/helm_tests/airflow_aux/test_airflow_common.py
b/chart/tests/helm_tests/airflow_aux/test_airflow_common.py
index 0a5b1556279..8c3418f7559 100644
--- a/chart/tests/helm_tests/airflow_aux/test_airflow_common.py
+++ b/chart/tests/helm_tests/airflow_aux/test_airflow_common.py
@@ -21,6 +21,8 @@ import pytest
import yaml
from chart_utils.helm_template_generator import render_chart
+METADATA_DB_VARS = {"AIRFLOW__DATABASE__SQL_ALCHEMY_CONN",
"AIRFLOW_CONN_AIRFLOW_DB"}
+
class TestAirflowCommon:
"""
@@ -348,10 +350,13 @@ class TestAirflowCommon:
expected_vars = [
"AIRFLOW_HOME",
"AIRFLOW__CORE__FERNET_KEY",
- "AIRFLOW_CONN_AIRFLOW_DB",
"AIRFLOW__CELERY__BROKER_URL",
+ "AIRFLOW_CONN_AIRFLOW_DB",
+ ]
+ # Workers only keep sql_alchemy_conn, disabled here, for the default
Celery result backend.
+ expected_vars_in_worker = ["DUMB_INIT_SETSID"] + [
+ var for var in expected_vars if var not in METADATA_DB_VARS
]
- expected_vars_in_worker = ["DUMB_INIT_SETSID"] + expected_vars
for doc in docs:
component = doc["metadata"]["labels"]["component"]
variables = expected_vars_in_worker if component == "worker" else
expected_vars
@@ -373,24 +378,26 @@ class TestAirflowCommon:
expected_vars_with_jwt = [
"AIRFLOW_HOME",
"AIRFLOW__CORE__FERNET_KEY",
- "AIRFLOW__DATABASE__SQL_ALCHEMY_CONN",
- "AIRFLOW_CONN_AIRFLOW_DB",
"AIRFLOW__API__SECRET_KEY",
"AIRFLOW__CELERY__BROKER_URL",
+ "AIRFLOW__DATABASE__SQL_ALCHEMY_CONN",
+ "AIRFLOW_CONN_AIRFLOW_DB",
"AIRFLOW__API_AUTH__JWT_SECRET",
]
expected_vars_no_jwt = [
"AIRFLOW_HOME",
"AIRFLOW__CORE__FERNET_KEY",
- "AIRFLOW__DATABASE__SQL_ALCHEMY_CONN",
- "AIRFLOW_CONN_AIRFLOW_DB",
"AIRFLOW__API__SECRET_KEY",
"AIRFLOW__CELERY__BROKER_URL",
+ "AIRFLOW__DATABASE__SQL_ALCHEMY_CONN",
+ "AIRFLOW_CONN_AIRFLOW_DB",
]
for doc in docs:
component = doc["metadata"]["labels"]["component"]
expected = expected_vars_with_jwt if component == "scheduler" else
expected_vars_no_jwt
- expected_in_worker = ["DUMB_INIT_SETSID"] + expected
+ expected_in_worker = ["DUMB_INIT_SETSID"] + [
+ var for var in expected if var != "AIRFLOW_CONN_AIRFLOW_DB"
+ ]
variables = expected_in_worker if component == "worker" else
expected
assert variables ==
jmespath.search("spec.template.spec.containers[0].env[*].name", doc), (
f"Wrong vars in {component}"
@@ -455,6 +462,176 @@ class TestAirflowCommon:
)
assert "AIRFLOW__API_AUTH__JWT_SECRET" not in env_names, f"Wrong
vars in {component}"
+ def
test_worker_keeps_sql_alchemy_conn_for_the_default_result_backend(self):
+ """Celery derives its result backend from sql_alchemy_conn when none
is configured."""
+ docs =
render_chart(show_only=["templates/workers/worker-deployment.yaml"])
+ names =
set(jmespath.search("spec.template.spec.containers[].env[].name", docs[0]) or
[])
+ assert names & METADATA_DB_VARS ==
{"AIRFLOW__DATABASE__SQL_ALCHEMY_CONN"}
+
+ @pytest.mark.parametrize(
+ "values",
+ [
+ pytest.param({"data": {"resultBackendSecretName":
"result-backend"}}, id="secret-name"),
+ pytest.param(
+ {
+ "data": {
+ "resultBackendConnection": {
+ "protocol": "postgresql",
+ "host": "results",
+ "port": 5432,
+ "user": "celery",
+ "pass": "x",
+ "db": "results",
+ "sslmode": "disable",
+ }
+ }
+ },
+ id="connection",
+ ),
+ pytest.param({"config": {"celery": {"result_backend":
"redis://r:6379/1"}}}, id="config"),
+ pytest.param({"config": {"celery": {"result_backend_cmd": "cat
/backend"}}}, id="config-cmd"),
+ pytest.param({"config": {"celery": {"result_backend_secret":
"backend"}}}, id="config-secret"),
+ ],
+ )
+ def test_metadata_db_env_absent_in_workers_with_a_result_backend(self,
values):
+ docs = render_chart(values=values,
show_only=["templates/workers/worker-deployment.yaml"])
+ names =
set(jmespath.search("spec.template.spec.containers[].env[].name", docs[0]) or
[])
+ assert not (names & METADATA_DB_VARS)
+
+ def
test_worker_keeps_sql_alchemy_conn_when_the_result_backend_secret_is_not_exposed(self):
+ docs = render_chart(
+ values={
+ "data": {"resultBackendSecretName": "result-backend"},
+ "enableBuiltInSecretEnvVars":
{"AIRFLOW__CELERY__RESULT_BACKEND": False},
+ },
+ show_only=["templates/workers/worker-deployment.yaml"],
+ )
+ names =
jmespath.search("spec.template.spec.containers[?name=='worker'].env[].name",
docs[0])
+ assert "AIRFLOW__DATABASE__SQL_ALCHEMY_CONN" in names
+
+ def test_worker_kerberos_sidecar_has_no_metadata_db_env_with_keda(self):
+ """KEDA reads the worker container's environment only."""
+ docs = render_chart(
+ values={
+ "airflowVersion": "3.4.0",
+ "workers": {"celery": {"keda": {"enabled": True},
"kerberosSidecar": {"enabled": True}}},
+ },
+ show_only=["templates/workers/worker-deployment.yaml"],
+ )
+ names = jmespath.search(
+
"spec.template.spec.initContainers[?name=='worker-kerberos'].env[].name",
docs[0]
+ )
+ assert names
+ assert not (set(names) & METADATA_DB_VARS)
+
+ @pytest.mark.parametrize(
+ ("values", "containers"),
+ [
+ pytest.param(
+ {"workers": {"celery": {"kerberosSidecar": {"enabled":
True}}}},
+ {"worker-kerberos"},
+ id="default-result-backend",
+ ),
+ pytest.param(
+ {
+ "data": {"resultBackendSecretName": "result-backend"},
+ "workers": {"celery": {"kerberosSidecar": {"enabled":
True}}},
+ },
+ {"worker", "worker-kerberos"},
+ id="separate-result-backend",
+ ),
+ ],
+ )
+ def
test_engine_args_cleared_in_worker_containers_without_the_connection(self,
values, containers):
+ """Those containers fall back to SQLite, which rejects engine args
meant for the real database."""
+ docs = render_chart(
+ values={
+ **values,
+ "airflowVersion": "3.4.0",
+ "config": {"database": {"sql_alchemy_engine_args":
'{"executemany_mode": "values"}'}},
+ },
+ show_only=["templates/workers/worker-deployment.yaml"],
+ )
+ pod_spec = docs[0]["spec"]["template"]["spec"]
+ cleared = {
+ container["name"]
+ for container in pod_spec["containers"] +
pod_spec["initContainers"]
+ if {"name": "AIRFLOW__DATABASE__SQL_ALCHEMY_ENGINE_ARGS", "value":
"{}"}
+ in container.get("env", [])
+ }
+ assert cleared == containers
+
+ @pytest.mark.parametrize(
+ ("values", "containers"),
+ [
+ pytest.param({}, {"worker-kerberos"}, id="default-result-backend"),
+ pytest.param(
+ {"data": {"resultBackendSecretName": "result-backend"}},
+ {"worker", "worker-kerberos"},
+ id="separate-result-backend",
+ ),
+ ],
+ )
+ def
test_migration_check_disabled_in_worker_containers_without_the_connection(self,
values, containers):
+ """CLI commands exit on the empty SQLite fallback when they check
migrations."""
+ docs = render_chart(
+ values={
+ **values,
+ "airflowVersion": "3.4.0",
+ "workers": {"celery": {"kerberosSidecar": {"enabled": True}}},
+ },
+ show_only=["templates/workers/worker-deployment.yaml"],
+ )
+ pod_spec = docs[0]["spec"]["template"]["spec"]
+ disabled = {
+ container["name"]
+ for container in pod_spec["containers"] +
pod_spec["initContainers"]
+ if {"name": "AIRFLOW__DATABASE__CHECK_MIGRATIONS", "value":
"False"} in container.get("env", [])
+ }
+ assert disabled == containers
+
+ def
test_worker_kerberos_sidecar_keeps_sql_alchemy_conn_before_airflow_3_4(self):
+ """`airflow kerberos` opens the metadata DB at startup there."""
+ docs = render_chart(
+ values={"airflowVersion": "3.3.2", "workers": {"celery":
{"kerberosSidecar": {"enabled": True}}}},
+ show_only=["templates/workers/worker-deployment.yaml"],
+ )
+ names = jmespath.search(
+
"spec.template.spec.initContainers[?name=='worker-kerberos'].env[].name",
docs[0]
+ )
+ assert set(names) & METADATA_DB_VARS ==
{"AIRFLOW__DATABASE__SQL_ALCHEMY_CONN"}
+ assert "AIRFLOW__DATABASE__CHECK_MIGRATIONS" not in names
+
+ def test_api_server_does_not_expose_the_airflow_db_connection(self):
+ """The Execution API answers task connection lookups from the API
server's environment."""
+ docs =
render_chart(show_only=["templates/api-server/api-server-deployment.yaml"])
+ names = jmespath.search("spec.template.spec.[containers,
initContainers][][].env[].name", docs[0])
+ assert "AIRFLOW__DATABASE__SQL_ALCHEMY_CONN" in names
+ assert "AIRFLOW_CONN_AIRFLOW_DB" not in names
+
+ def test_worker_migration_init_container_has_metadata_db_env(self):
+ """The wait-for-migration init container queries the DB."""
+ docs =
render_chart(show_only=["templates/workers/worker-deployment.yaml"])
+ assert "AIRFLOW__DATABASE__SQL_ALCHEMY_CONN" in jmespath.search(
+
"spec.template.spec.initContainers[?name=='wait-for-airflow-migrations'].env[].name",
+ docs[0],
+ )
+
+ def test_metadata_db_env_in_all_required_components(self):
+ docs = render_chart(
+ show_only=[
+ "templates/scheduler/scheduler-deployment.yaml",
+ "templates/api-server/api-server-deployment.yaml",
+ "templates/dag-processor/dag-processor-deployment.yaml",
+ "templates/triggerer/triggerer-deployment.yaml",
+ ],
+ )
+ assert docs
+ for doc in docs:
+ component = doc["metadata"]["labels"]["component"]
+ names =
jmespath.search("spec.template.spec.containers[0].env[*].name", doc)
+ assert "AIRFLOW__DATABASE__SQL_ALCHEMY_CONN" in names,
f"{component} lost its DB connection"
+
def test_have_all_config_mounts_on_init_containers(self):
docs = render_chart(
show_only=[
diff --git a/chart/tests/helm_tests/airflow_aux/test_pod_template_file.py
b/chart/tests/helm_tests/airflow_aux/test_pod_template_file.py
index 3079dd1cf68..02c46c58521 100644
--- a/chart/tests/helm_tests/airflow_aux/test_pod_template_file.py
+++ b/chart/tests/helm_tests/airflow_aux/test_pod_template_file.py
@@ -56,6 +56,82 @@ class TestPodTemplateFile:
assert jmespath.search("spec.containers[0].image", docs[0]) is not None
assert jmespath.search("spec.containers[0].name", docs[0]) == "base"
+ def test_should_not_carry_metadata_db_credentials(self):
+ """KubernetesExecutor task pods run Dag-author code and must not hold
DB credentials.
+
+ The task SDK reaches the Execution API rather than the metadata
database, so the
+ task pod has no functional need for the connection string. Unlike
Celery workers
+ there is no KEDA exception here: KEDA does not scale task pods.
+ """
+ docs = render_chart(
+ values={
+ "airflowVersion": "3.4.0",
+ "workers": {"kubernetes": {"kerberosSidecar": {"enabled":
True}}},
+ },
+ show_only=["templates/pod-template-file.yaml"],
+ chart_dir=self.temp_chart_dir,
+ )
+ metadata_db_vars = {"AIRFLOW__DATABASE__SQL_ALCHEMY_CONN",
"AIRFLOW_CONN_AIRFLOW_DB"}
+ spec = docs[0]["spec"]
+ assert "worker-kerberos" in [container["name"] for container in
spec["initContainers"]]
+ for container in spec["containers"] + spec["initContainers"]:
+ names = {env["name"] for env in container.get("env", [])}
+ assert not (names & metadata_db_vars), (
+ f"metadata DB env present in task-pod container
{container['name']}"
+ )
+
+ @pytest.mark.parametrize(
+ "option",
+ ["sql_alchemy_engine_args", "sql_alchemy_engine_args_cmd",
"sql_alchemy_engine_args_secret"],
+ )
+ def test_engine_args_cleared_in_task_pods(self, option):
+ """Task pods fall back to SQLite, which rejects engine args meant for
the real database."""
+ docs = render_chart(
+ values={
+ "airflowVersion": "3.4.0",
+ "config": {"database": {option: '{"executemany_mode":
"values"}'}},
+ "workers": {"kubernetes": {"kerberosSidecar": {"enabled":
True}}},
+ },
+ show_only=["templates/pod-template-file.yaml"],
+ chart_dir=self.temp_chart_dir,
+ )
+ spec = docs[0]["spec"]
+ for name, containers in (("base", spec["containers"]),
("worker-kerberos", spec["initContainers"])):
+ env = next(container["env"] for container in containers if
container["name"] == name)
+ assert {"name": "AIRFLOW__DATABASE__SQL_ALCHEMY_ENGINE_ARGS",
"value": "{}"} in env
+
+ def test_migration_check_disabled_in_task_pods(self):
+ """CLI commands exit on the empty SQLite fallback when they check
migrations."""
+ docs = render_chart(
+ values={
+ "airflowVersion": "3.4.0",
+ "workers": {"kubernetes": {"kerberosSidecar": {"enabled":
True}}},
+ },
+ show_only=["templates/pod-template-file.yaml"],
+ chart_dir=self.temp_chart_dir,
+ )
+ env =
jmespath.search("spec.initContainers[?name=='worker-kerberos'].env[]", docs[0])
+ assert {"name": "AIRFLOW__DATABASE__CHECK_MIGRATIONS", "value":
"False"} in env
+
+ def test_kerberos_sidecar_keeps_sql_alchemy_conn_before_airflow_3_4(self):
+ """`airflow kerberos` opens the metadata DB at startup there."""
+ docs = render_chart(
+ values={
+ "airflowVersion": "3.3.2",
+ "workers": {"kubernetes": {"kerberosSidecar": {"enabled":
True}}},
+ },
+ show_only=["templates/pod-template-file.yaml"],
+ chart_dir=self.temp_chart_dir,
+ )
+ names =
jmespath.search("spec.initContainers[?name=='worker-kerberos'].env[].name",
docs[0])
+ assert "AIRFLOW__DATABASE__SQL_ALCHEMY_CONN" in names
+ assert "AIRFLOW_CONN_AIRFLOW_DB" not in names
+
+ def test_engine_args_untouched_when_not_configured(self):
+ docs = render_chart(show_only=["templates/pod-template-file.yaml"],
chart_dir=self.temp_chart_dir)
+ names = jmespath.search("spec.containers[0].env[].name", docs[0])
+ assert "AIRFLOW__DATABASE__SQL_ALCHEMY_ENGINE_ARGS" not in names
+
def test_should_add_an_init_container_if_git_sync_is_true(self):
docs = render_chart(
values={
