jedcunningham commented on a change in pull request #18974:
URL: https://github.com/apache/airflow/pull/18974#discussion_r770216448
##########
File path: chart/values.schema.json
##########
@@ -711,6 +711,58 @@
}
]
},
+ "enableBuiltInSecretEnvVars": {
+ "description": "Uses built-in secret values set as environment
variables passed to Airflow. You should supply corresponding environment
variables as ``extraEnv`` variables if you disable them here.",
+ "type": "object",
Review comment:
```suggestion
"type": "object",
"additionalProperties": false,
```
##########
File path: chart/values.schema.json
##########
@@ -711,6 +711,58 @@
}
]
},
+ "enableBuiltInSecretEnvVars": {
+ "description": "Uses built-in secret values set as environment
variables passed to Airflow. You should supply corresponding environment
variables as ``extraEnv`` variables if you disable them here.",
+ "type": "object",
+ "x-docsSection": "Airflow",
+ "properties": {
+ "AIRFLOW__CORE_FERNET_KEY": {
+ "description": "Enable Airflow fernet key variable read
from .Values.fernetKeySecretName",
+ "type": "boolean",
+ "default": true
+ },
+ "AIRFLOW__CORE__SQL_ALCHEMY_CONN": {
+ "description": "Enable Airflow meta-data connection
variable read from .Values.data.metadataSecretName",
+ "type": "boolean",
+ "default": true
+ },
+ "AIRFLOW_CONN_AIRFLOW_DB": {
+ "description": "Enable Airflow meta-data connection
variable read from .Values.data.metadataSecretName (Airflow 1.10 variant)",
+ "type": "boolean",
+ "default": true
+ },
+ "AIRFLOW__WEBSERVER__SECRET_KEY": {
+ "description": "Enable Webserver secret key variable read
from .Values.webserverSecretKeySecretName",
+ "type": "boolean",
+ "default": true
+ },
+ "AIRFLOW__CELERY__CELERY_RESULT_BACKEND": {
+ "description": "Enable Celery backend url variable read
from .Values.data.resultBackendSecretName",
+ "type": "boolean",
+ "default": true
+ },
+ "AIRFLOW__CELERY__RESULT_BACKEND": {
+ "description": "Enable Celery backend url variable read
from .Values.data.resultBackendSecretName (Airflow 1.10 variant)",
+ "type": "boolean",
+ "default": true
+ },
+ "AIRFLOW__CELERY__BROKER_URL": {
+ "description": "Enable Celery broker url variable read
from .Values.data.brokerUrlSecretName",
+ "type": "boolean",
+ "default": true
+ },
+ "AIRFLOW__ELASTICSEARCH__HOST": {
+ "description": "Elasticsearch Host variable read from
.Values.elasticsearch.secretName",
+ "type": "boolean",
+ "default": true
+ },
+ "AIRFLOW__ELASTICSEARCH__ELASTICSEARCH_HOST": {
+ "description": "Elasticsearch Host variable read from
.Values.data.metadataSecretName (Airflow 1.10 variant)",
Review comment:
```suggestion
"description": "Elasticsearch Host variable read from
.Values.data.metadataSecretName (Airflow <1.10.4 variant)",
```
This changed in 1.10.4.
##########
File path: chart/templates/_helpers.yaml
##########
@@ -37,56 +37,74 @@ If release name contains chart name it will be used as a
full name.
{{/* Standard Airflow environment variables */}}
{{- define "standard_airflow_environment" }}
# Hard Coded Airflow Envs
+ {{- if .Values.enableBuiltInSecretEnvVars.AIRFLOW_CORE_FERNET_KEY }}
- name: AIRFLOW__CORE__FERNET_KEY
valueFrom:
secretKeyRef:
name: {{ template "fernet_key_secret" . }}
key: fernet-key
+ {{- end }}
+ {{- if .Values.enableBuiltInSecretEnvVars.AIRFLOW__CORE__SQL_ALCHEMY_CONN }}
- name: AIRFLOW__CORE__SQL_ALCHEMY_CONN
valueFrom:
secretKeyRef:
name: {{ template "airflow_metadata_secret" . }}
key: connection
+ {{- end }}
+ {{- if .Values.enableBuiltInSecretEnvVars.AIRFLOW_CONN_AIRFLOW_DB }}
- name: AIRFLOW_CONN_AIRFLOW_DB
valueFrom:
secretKeyRef:
name: {{ template "airflow_metadata_secret" . }}
key: connection
+ {{- end }}
+ {{- if .Values.enableBuiltInSecretEnvVars.AIRFLOW__WEBSERVER__SECRET_KEY }}
- name: AIRFLOW__WEBSERVER__SECRET_KEY
valueFrom:
secretKeyRef:
name: {{ template "webserver_secret_key_secret" . }}
key: webserver-secret-key
+ {{- end }}
{{- if or (eq .Values.executor "CeleryExecutor") (eq .Values.executor
"CeleryKubernetesExecutor") }}
+ {{- if
.Values.enableBuiltInSecretEnvVars.AIRFLOW__CELERY__CELERY_RESULT_BACKEND }}
- name: AIRFLOW__CELERY__CELERY_RESULT_BACKEND
Review comment:
```suggestion
# (Airflow <1.10.0 variant)
- name: AIRFLOW__CELERY__CELERY_RESULT_BACKEND
```
Might be nice to include why this is included here as well.
##########
File path: chart/values.schema.json
##########
@@ -711,6 +711,58 @@
}
]
},
+ "enableBuiltInSecretEnvVars": {
+ "description": "Uses built-in secret values set as environment
variables passed to Airflow. You should supply corresponding environment
variables as ``extraEnv`` variables if you disable them here.",
+ "type": "object",
+ "x-docsSection": "Airflow",
+ "properties": {
+ "AIRFLOW__CORE_FERNET_KEY": {
+ "description": "Enable Airflow fernet key variable read
from .Values.fernetKeySecretName",
Review comment:
I'm a little torn on the pattern of mentioning
`.Values.fernetKeySecretName`, because by default that is null and the secret
name is defaulted to `{{ .Release.Name }}-fernet-key` in this specific case. I
wonder if this would be better?
```suggestion
"description": "Enable Airflow fernet key variable read
from fernet key secret",
```
##########
File path: chart/values.schema.json
##########
@@ -711,6 +711,58 @@
}
]
},
+ "enableBuiltInSecretEnvVars": {
+ "description": "Uses built-in secret values set as environment
variables passed to Airflow. You should supply corresponding environment
variables as ``extraEnv`` variables if you disable them here.",
+ "type": "object",
+ "x-docsSection": "Airflow",
+ "properties": {
+ "AIRFLOW__CORE_FERNET_KEY": {
+ "description": "Enable Airflow fernet key variable read
from .Values.fernetKeySecretName",
+ "type": "boolean",
+ "default": true
+ },
+ "AIRFLOW__CORE__SQL_ALCHEMY_CONN": {
+ "description": "Enable Airflow meta-data connection
variable read from .Values.data.metadataSecretName",
+ "type": "boolean",
+ "default": true
+ },
+ "AIRFLOW_CONN_AIRFLOW_DB": {
+ "description": "Enable Airflow meta-data connection
variable read from .Values.data.metadataSecretName (Airflow 1.10 variant)",
+ "type": "boolean",
+ "default": true
+ },
+ "AIRFLOW__WEBSERVER__SECRET_KEY": {
+ "description": "Enable Webserver secret key variable read
from .Values.webserverSecretKeySecretName",
+ "type": "boolean",
+ "default": true
+ },
+ "AIRFLOW__CELERY__CELERY_RESULT_BACKEND": {
+ "description": "Enable Celery backend url variable read
from .Values.data.resultBackendSecretName",
+ "type": "boolean",
+ "default": true
+ },
+ "AIRFLOW__CELERY__RESULT_BACKEND": {
+ "description": "Enable Celery backend url variable read
from .Values.data.resultBackendSecretName (Airflow 1.10 variant)",
+ "type": "boolean",
+ "default": true
+ },
+ "AIRFLOW__CELERY__BROKER_URL": {
+ "description": "Enable Celery broker url variable read
from .Values.data.brokerUrlSecretName",
+ "type": "boolean",
+ "default": true
+ },
+ "AIRFLOW__ELASTICSEARCH__HOST": {
+ "description": "Elasticsearch Host variable read from
.Values.elasticsearch.secretName",
+ "type": "boolean",
+ "default": true
+ },
+ "AIRFLOW__ELASTICSEARCH__ELASTICSEARCH_HOST": {
+ "description": "Elasticsearch Host variable read from
.Values.data.metadataSecretName (Airflow 1.10 variant)",
Review comment:
```suggestion
"description": "Enable Elasticsearch Host variable read
from .Values.data.metadataSecretName (Airflow 1.10 variant)",
```
##########
File path: docs/helm-chart/production-guide.rst
##########
@@ -283,3 +283,52 @@ In order to enable the usage of SCCs, one must set the
parameter :ref:`rbac.crea
In this chart, SCCs are bound to the Pods via RoleBindings meaning that the
option ``rbac.create`` must also be set to ``true`` in order to fully enable
the SCC usage.
For more information about SCCs and what can be achieved with this construct,
please refer to `Managing security context constraints
<https://docs.openshift.com/container-platform/latest/authentication/managing-security-context-constraints.html#scc-prioritization_configuring-internal-oauth/>`_.
+
+Environment variables and secrets
+---------------------------------
+
+The Helm Chart by default uses Kubernetes Secrets to store secrets that are
needed by Airflow.
+Content of those secrets is by default turned into environment variables that
are read by
+Airflow (some of the environment variables have several variants to support
older versions of Airflow)
+
++------------------------------------------+------------------------------------------------+
+| Secret name in configuration | Airflow Environment Variable
|
++==========================================+================================================+
+| ``.Values.data.metadataSecretName`` | ``AIRFLOW_CONN_AIRFLOW_DB``
|
++------------------------------------------+------------------------------------------------+
+| ``.Values.fernetKeySecretName`` | ``AIRFLOW__CORE_FERNET_KEY``
|
++------------------------------------------+------------------------------------------------+
+| ``.Values.data.metadataSecretName`` |
``AIRFLOW__CORE__SQL_ALCHEMY_CONN,`` |
++------------------------------------------+------------------------------------------------+
+| ``.Values.webserverSecretKeySecretName`` |
``AIRFLOW__WEBSERVER__SECRET_KEY`` |
++------------------------------------------+------------------------------------------------+
+| ``.Values.data.resultBackendSecretName`` |
``AIRFLOW__CELERY__CELERY_RESULT_BACKEND`` |
++------------------------------------------+------------------------------------------------+
+| ``.Values.data.resultBackendSecretName`` |
``AIRFLOW__CELERY__RESULT_BACKEND`` |
++------------------------------------------+------------------------------------------------+
+| ``.Values.data.brokerUrlSecretName`` | ``AIRFLOW__CELERY__BROKER_URL``
|
++------------------------------------------+------------------------------------------------+
+| ``.Values.elasticsearch.secretName`` | ``AIRFLOW__ELASTICSEARCH__HOST``
|
++------------------------------------------+------------------------------------------------+
+| ``.Values.elasticsearch.secretName`` |
``AIRFLOW__ELASTICSEARCH__ELASTICSEARCH_HOST`` |
++------------------------------------------+------------------------------------------------+
+
+However Airflow supports other ways of setting secret values - for example you
can specify a system
+command to retrieve and automatically rotate the secret (by defining variable
with _CMD suffix) or
+to retrieve a variable from secret backed (by defining the variable with
_SECRET suffix). In those
+cases you should disable the built in variables retrieved from Kubernetes
secrets, by specifying
+``.Values.enableBuiltInSecretEnvVars.<VARIABLE_NAME>`` to false and adding the
_CMD or _SECRET
+variable as ``extraEnv`` variable.
+
+For example in order to use a command to retrieve the DB connection you should
(in your ``values.yaml``
+file) specify:
+
+.. code-block:: yaml
+
+ extraEnv:
+ AIRFLOW_CONN_AIRFLOW_DB_CMD: "/usr/local/bin/retrieve_connection_url"
+ enableBuiltInSecretEnvVars:
+ AIRFLOW_CONN_AIRFLOW_DB: false
+
+You can read more about advanced ways of setting configuration variables in the
+:doc:`apache-airflow:howto/set-config`
Review comment:
```suggestion
You can read more about advanced ways of setting configuration variables in
:doc:`apache-airflow:howto/set-config`.
```
##########
File path: chart/values.schema.json
##########
@@ -711,6 +711,58 @@
}
]
},
+ "enableBuiltInSecretEnvVars": {
+ "description": "Uses built-in secret values set as environment
variables passed to Airflow. You should supply corresponding environment
variables as ``extraEnv`` variables if you disable them here.",
+ "type": "object",
+ "x-docsSection": "Airflow",
+ "properties": {
+ "AIRFLOW__CORE_FERNET_KEY": {
+ "description": "Enable Airflow fernet key variable read
from .Values.fernetKeySecretName",
+ "type": "boolean",
+ "default": true
+ },
+ "AIRFLOW__CORE__SQL_ALCHEMY_CONN": {
+ "description": "Enable Airflow meta-data connection
variable read from .Values.data.metadataSecretName",
+ "type": "boolean",
+ "default": true
+ },
+ "AIRFLOW_CONN_AIRFLOW_DB": {
+ "description": "Enable Airflow meta-data connection
variable read from .Values.data.metadataSecretName (Airflow 1.10 variant)",
Review comment:
```suggestion
"description": "Enable Airflow meta-data connection
variable read from .Values.data.metadataSecretName",
```
##########
File path: docs/helm-chart/production-guide.rst
##########
@@ -283,3 +283,52 @@ In order to enable the usage of SCCs, one must set the
parameter :ref:`rbac.crea
In this chart, SCCs are bound to the Pods via RoleBindings meaning that the
option ``rbac.create`` must also be set to ``true`` in order to fully enable
the SCC usage.
For more information about SCCs and what can be achieved with this construct,
please refer to `Managing security context constraints
<https://docs.openshift.com/container-platform/latest/authentication/managing-security-context-constraints.html#scc-prioritization_configuring-internal-oauth/>`_.
+
+Environment variables and secrets
+---------------------------------
+
+The Helm Chart by default uses Kubernetes Secrets to store secrets that are
needed by Airflow.
+Content of those secrets is by default turned into environment variables that
are read by
+Airflow (some of the environment variables have several variants to support
older versions of Airflow)
Review comment:
```suggestion
Airflow (some of the environment variables have several variants to support
older versions of Airflow).
```
Or maybe `:`?
##########
File path: chart/values.schema.json
##########
@@ -711,6 +711,58 @@
}
]
},
+ "enableBuiltInSecretEnvVars": {
+ "description": "Uses built-in secret values set as environment
variables passed to Airflow. You should supply corresponding environment
variables as ``extraEnv`` variables if you disable them here.",
+ "type": "object",
+ "x-docsSection": "Airflow",
+ "properties": {
+ "AIRFLOW__CORE_FERNET_KEY": {
+ "description": "Enable Airflow fernet key variable read
from .Values.fernetKeySecretName",
+ "type": "boolean",
+ "default": true
+ },
+ "AIRFLOW__CORE__SQL_ALCHEMY_CONN": {
+ "description": "Enable Airflow meta-data connection
variable read from .Values.data.metadataSecretName",
+ "type": "boolean",
+ "default": true
+ },
+ "AIRFLOW_CONN_AIRFLOW_DB": {
+ "description": "Enable Airflow meta-data connection
variable read from .Values.data.metadataSecretName (Airflow 1.10 variant)",
+ "type": "boolean",
+ "default": true
+ },
+ "AIRFLOW__WEBSERVER__SECRET_KEY": {
+ "description": "Enable Webserver secret key variable read
from .Values.webserverSecretKeySecretName",
+ "type": "boolean",
+ "default": true
+ },
+ "AIRFLOW__CELERY__CELERY_RESULT_BACKEND": {
+ "description": "Enable Celery backend url variable read
from .Values.data.resultBackendSecretName",
+ "type": "boolean",
+ "default": true
+ },
+ "AIRFLOW__CELERY__RESULT_BACKEND": {
+ "description": "Enable Celery backend url variable read
from .Values.data.resultBackendSecretName (Airflow 1.10 variant)",
+ "type": "boolean",
+ "default": true
+ },
+ "AIRFLOW__CELERY__BROKER_URL": {
+ "description": "Enable Celery broker url variable read
from .Values.data.brokerUrlSecretName",
+ "type": "boolean",
+ "default": true
+ },
+ "AIRFLOW__ELASTICSEARCH__HOST": {
+ "description": "Elasticsearch Host variable read from
.Values.elasticsearch.secretName",
Review comment:
```suggestion
"description": "Enable Elasticsearch Host variable read
from .Values.elasticsearch.secretName",
```
Just to be consistent.
##########
File path: chart/values.schema.json
##########
@@ -711,6 +711,58 @@
}
]
},
+ "enableBuiltInSecretEnvVars": {
+ "description": "Uses built-in secret values set as environment
variables passed to Airflow. You should supply corresponding environment
variables as ``extraEnv`` variables if you disable them here.",
+ "type": "object",
+ "x-docsSection": "Airflow",
+ "properties": {
+ "AIRFLOW__CORE_FERNET_KEY": {
+ "description": "Enable Airflow fernet key variable read
from .Values.fernetKeySecretName",
+ "type": "boolean",
+ "default": true
+ },
+ "AIRFLOW__CORE__SQL_ALCHEMY_CONN": {
+ "description": "Enable Airflow meta-data connection
variable read from .Values.data.metadataSecretName",
+ "type": "boolean",
+ "default": true
+ },
+ "AIRFLOW_CONN_AIRFLOW_DB": {
+ "description": "Enable Airflow meta-data connection
variable read from .Values.data.metadataSecretName (Airflow 1.10 variant)",
+ "type": "boolean",
+ "default": true
+ },
+ "AIRFLOW__WEBSERVER__SECRET_KEY": {
+ "description": "Enable Webserver secret key variable read
from .Values.webserverSecretKeySecretName",
+ "type": "boolean",
+ "default": true
+ },
+ "AIRFLOW__CELERY__CELERY_RESULT_BACKEND": {
+ "description": "Enable Celery backend url variable read
from .Values.data.resultBackendSecretName",
+ "type": "boolean",
+ "default": true
+ },
+ "AIRFLOW__CELERY__RESULT_BACKEND": {
+ "description": "Enable Celery backend url variable read
from .Values.data.resultBackendSecretName (Airflow 1.10 variant)",
Review comment:
```suggestion
"description": "Enable Celery backend url variable read
from .Values.data.resultBackendSecretName",
```
This isn't the earlier variant, the other one is.
##########
File path: chart/values.schema.json
##########
@@ -711,6 +711,58 @@
}
]
},
+ "enableBuiltInSecretEnvVars": {
+ "description": "Uses built-in secret values set as environment
variables passed to Airflow. You should supply corresponding environment
variables as ``extraEnv`` variables if you disable them here.",
+ "type": "object",
+ "x-docsSection": "Airflow",
+ "properties": {
+ "AIRFLOW__CORE_FERNET_KEY": {
+ "description": "Enable Airflow fernet key variable read
from .Values.fernetKeySecretName",
+ "type": "boolean",
+ "default": true
+ },
+ "AIRFLOW__CORE__SQL_ALCHEMY_CONN": {
+ "description": "Enable Airflow meta-data connection
variable read from .Values.data.metadataSecretName",
+ "type": "boolean",
+ "default": true
+ },
+ "AIRFLOW_CONN_AIRFLOW_DB": {
+ "description": "Enable Airflow meta-data connection
variable read from .Values.data.metadataSecretName (Airflow 1.10 variant)",
+ "type": "boolean",
+ "default": true
+ },
+ "AIRFLOW__WEBSERVER__SECRET_KEY": {
+ "description": "Enable Webserver secret key variable read
from .Values.webserverSecretKeySecretName",
+ "type": "boolean",
+ "default": true
+ },
+ "AIRFLOW__CELERY__CELERY_RESULT_BACKEND": {
+ "description": "Enable Celery backend url variable read
from .Values.data.resultBackendSecretName",
Review comment:
```suggestion
"description": "Enable Celery backend url variable read
from .Values.data.resultBackendSecretName (Airflow <1.10.0 variant)",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
