jedcunningham commented on a change in pull request #16059: URL: https://github.com/apache/airflow/pull/16059#discussion_r638918590
########## File path: docs/helm-chart/adding-connections-and-variables.rst ########## @@ -0,0 +1,83 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + + +Adding Connections, Secrets and Environment Variables +================================================== + +You can programmatically add Connections, Secrets and Environment Variables to your +airflow deployment using Helm chart. Review comment: ```suggestion You can programmatically add Connections, Variables and arbitrary Environment Variables to your Airflow deployment using the Helm chart. ``` I'd probably leave the k8s secret piece out of the mix here, personally. ########## File path: docs/helm-chart/adding-connections-and-variables.rst ########## @@ -0,0 +1,83 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + + +Adding Connections, Secrets and Environment Variables +================================================== + +You can programmatically add Connections, Secrets and Environment Variables to your +airflow deployment using Helm chart. + + +Connections and Secrets +----------------------- +Under the ``secret`` and ``extraSecret`` sections of the ``values.yaml`` you can pass connection strings and secrets +into Airflow using Helm chart. To illustrate, lets create a yaml file to override values under these sections of the +``values.yaml`` file. + +.. code-block:: yaml + + secret: + - envName: "my-env" + secretName: "my-secret-name" + secretKey: "my-secret-key" + + + extraSecrets: + {{ .Release.Name }}-airflow-connections: + data: | + AIRFLOW_CONN_GCP: 'base64_encoded_gcp_conn_string' + AIRFLOW_CONN_AWS: 'base64_encoded_aws_conn_string' + stringData: | + AIRFLOW_CONN_OTHER: 'other_conn' + {{ .Release.Name }}-other-secret-name-suffix: | + data: | + + +Variables +--------- +Airflow supports Variables which enable users craft dynamic DAGs. You can set variables in Airflow in three ways - UI, +commandline, and within your DAG file. See:doc:`/howto/variable`.` for more. + +With Helm chart, you can also inject environment variables into airflow. For example, in a yaml file to Review comment: ```suggestion With the Helm chart, you can also inject environment variables into Airflow. For example, in a yaml file to ``` ########## File path: docs/helm-chart/adding-connections-and-variables.rst ########## @@ -0,0 +1,83 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + + +Adding Connections, Secrets and Environment Variables +================================================== + +You can programmatically add Connections, Secrets and Environment Variables to your +airflow deployment using Helm chart. + + +Connections and Secrets +----------------------- +Under the ``secret`` and ``extraSecret`` sections of the ``values.yaml`` you can pass connection strings and secrets +into Airflow using Helm chart. To illustrate, lets create a yaml file to override values under these sections of the +``values.yaml`` file. + +.. code-block:: yaml + + secret: + - envName: "my-env" + secretName: "my-secret-name" + secretKey: "my-secret-key" + + + extraSecrets: + {{ .Release.Name }}-airflow-connections: + data: | + AIRFLOW_CONN_GCP: 'base64_encoded_gcp_conn_string' + AIRFLOW_CONN_AWS: 'base64_encoded_aws_conn_string' + stringData: | + AIRFLOW_CONN_OTHER: 'other_conn' + {{ .Release.Name }}-other-secret-name-suffix: | + data: | + + +Variables +--------- +Airflow supports Variables which enable users craft dynamic DAGs. You can set variables in Airflow in three ways - UI, Review comment: ```suggestion Airflow supports Variables which enable users to craft dynamic DAGs. You can set Variables in Airflow in three ways - UI, ``` ########## File path: docs/helm-chart/adding-connections-and-variables.rst ########## @@ -0,0 +1,83 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + + +Adding Connections, Secrets and Environment Variables +================================================== + +You can programmatically add Connections, Secrets and Environment Variables to your +airflow deployment using Helm chart. + + +Connections and Secrets +----------------------- +Under the ``secret`` and ``extraSecret`` sections of the ``values.yaml`` you can pass connection strings and secrets +into Airflow using Helm chart. To illustrate, lets create a yaml file to override values under these sections of the +``values.yaml`` file. + +.. code-block:: yaml + + secret: + - envName: "my-env" + secretName: "my-secret-name" + secretKey: "my-secret-key" + + + extraSecrets: + {{ .Release.Name }}-airflow-connections: + data: | + AIRFLOW_CONN_GCP: 'base64_encoded_gcp_conn_string' + AIRFLOW_CONN_AWS: 'base64_encoded_aws_conn_string' + stringData: | + AIRFLOW_CONN_OTHER: 'other_conn' + {{ .Release.Name }}-other-secret-name-suffix: | + data: | + + +Variables +--------- +Airflow supports Variables which enable users craft dynamic DAGs. You can set variables in Airflow in three ways - UI, +commandline, and within your DAG file. See:doc:`/howto/variable`.` for more. Review comment: ```suggestion commandline, and within your DAG file. See :doc:`/howto/variable` for more. ``` I think this is the right format for the link? ########## File path: docs/helm-chart/adding-connections-and-variables.rst ########## @@ -0,0 +1,83 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + + +Adding Connections, Secrets and Environment Variables +================================================== + +You can programmatically add Connections, Secrets and Environment Variables to your +airflow deployment using Helm chart. + + +Connections and Secrets +----------------------- +Under the ``secret`` and ``extraSecret`` sections of the ``values.yaml`` you can pass connection strings and secrets +into Airflow using Helm chart. To illustrate, lets create a yaml file to override values under these sections of the Review comment: ```suggestion Under the ``secret`` and ``extraSecret`` sections of the ``values.yaml`` you can pass connection strings and sensitive environemt variables into Airflow using the Helm chart. To illustrate, lets create a yaml file to override values under these sections of the ``` ########## File path: docs/helm-chart/adding-connections-and-variables.rst ########## @@ -0,0 +1,83 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + + +Adding Connections, Secrets and Environment Variables +================================================== + +You can programmatically add Connections, Secrets and Environment Variables to your +airflow deployment using Helm chart. + + +Connections and Secrets +----------------------- +Under the ``secret`` and ``extraSecret`` sections of the ``values.yaml`` you can pass connection strings and secrets +into Airflow using Helm chart. To illustrate, lets create a yaml file to override values under these sections of the +``values.yaml`` file. + +.. code-block:: yaml + + secret: + - envName: "my-env" + secretName: "my-secret-name" + secretKey: "my-secret-key" + + + extraSecrets: + {{ .Release.Name }}-airflow-connections: + data: | + AIRFLOW_CONN_GCP: 'base64_encoded_gcp_conn_string' + AIRFLOW_CONN_AWS: 'base64_encoded_aws_conn_string' + stringData: | + AIRFLOW_CONN_OTHER: 'other_conn' + {{ .Release.Name }}-other-secret-name-suffix: | + data: | + + +Variables +--------- +Airflow supports Variables which enable users craft dynamic DAGs. You can set variables in Airflow in three ways - UI, +commandline, and within your DAG file. See:doc:`/howto/variable`.` for more. + +With Helm chart, you can also inject environment variables into airflow. For example, in a yaml file to +override values of interest in the ``env`` section of the ``values.yaml`` file. + +.. code-block:: yaml + + env: + - name: "AIRFLOW_VAR_KEY" + value: "value_1" + + - name: "AIRFLOW_VAR_ANOTHER_KEY" + value: "value_2" + + +You can also utilize the ``extraEnv`` and ``extraEnvFrom`` to set more possibly "templatable" values for your airflow +deployment. + +.. code-block:: yaml + + extraEnv: | + - name: AIRFLOW__CORE__LOAD_EXAMPLES + value: 'True' Review comment: ```suggestion - name: AIRFLOW_VAR_HELM_RELEASE_NAME value: '{{ .Release.Name }}' ``` Might as well have an example that uses templating. ########## File path: docs/helm-chart/adding-connections-and-variables.rst ########## @@ -0,0 +1,83 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + + +Adding Connections, Secrets and Environment Variables +================================================== + +You can programmatically add Connections, Secrets and Environment Variables to your +airflow deployment using Helm chart. + + +Connections and Secrets +----------------------- +Under the ``secret`` and ``extraSecret`` sections of the ``values.yaml`` you can pass connection strings and secrets +into Airflow using Helm chart. To illustrate, lets create a yaml file to override values under these sections of the +``values.yaml`` file. + +.. code-block:: yaml + + secret: + - envName: "my-env" + secretName: "my-secret-name" + secretKey: "my-secret-key" + + + extraSecrets: + {{ .Release.Name }}-airflow-connections: + data: | + AIRFLOW_CONN_GCP: 'base64_encoded_gcp_conn_string' + AIRFLOW_CONN_AWS: 'base64_encoded_aws_conn_string' + stringData: | + AIRFLOW_CONN_OTHER: 'other_conn' + {{ .Release.Name }}-other-secret-name-suffix: | + data: | + + +Variables +--------- +Airflow supports Variables which enable users craft dynamic DAGs. You can set variables in Airflow in three ways - UI, +commandline, and within your DAG file. See:doc:`/howto/variable`.` for more. + +With Helm chart, you can also inject environment variables into airflow. For example, in a yaml file to +override values of interest in the ``env`` section of the ``values.yaml`` file. + +.. code-block:: yaml + + env: + - name: "AIRFLOW_VAR_KEY" + value: "value_1" + + - name: "AIRFLOW_VAR_ANOTHER_KEY" + value: "value_2" + + +You can also utilize the ``extraEnv`` and ``extraEnvFrom`` to set more possibly "templatable" values for your airflow +deployment. Review comment: ```suggestion You can also utilize ``extraEnv`` and ``extraEnvFrom`` if you need the name or value to be templated. ``` ########## File path: docs/helm-chart/adding-connections-and-variables.rst ########## @@ -0,0 +1,83 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + + +Adding Connections, Secrets and Environment Variables +================================================== + +You can programmatically add Connections, Secrets and Environment Variables to your +airflow deployment using Helm chart. + + +Connections and Secrets +----------------------- +Under the ``secret`` and ``extraSecret`` sections of the ``values.yaml`` you can pass connection strings and secrets +into Airflow using Helm chart. To illustrate, lets create a yaml file to override values under these sections of the +``values.yaml`` file. + +.. code-block:: yaml + + secret: + - envName: "my-env" + secretName: "my-secret-name" + secretKey: "my-secret-key" + + + extraSecrets: + {{ .Release.Name }}-airflow-connections: + data: | + AIRFLOW_CONN_GCP: 'base64_encoded_gcp_conn_string' + AIRFLOW_CONN_AWS: 'base64_encoded_aws_conn_string' + stringData: | + AIRFLOW_CONN_OTHER: 'other_conn' + {{ .Release.Name }}-other-secret-name-suffix: | + data: | Review comment: ```suggestion secret: - envName: "AIRFLOW_CONN_GCP" secretName: "my-airflow-connections" secretKey: "AIRFLOW_CONN_GCP" - envName: "my-env" secretName: "my-secret-name" secretKey: "my-secret-key" extraSecrets: my-airflow-connections: data: | AIRFLOW_CONN_GCP: 'base64_encoded_gcp_conn_string' my-secret-name: | stringData: | my-secret-key: my-secret ``` Have it be a working example, and show both an env var and a connection. ########## File path: docs/helm-chart/adding-connections-and-variables.rst ########## @@ -0,0 +1,83 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + + +Adding Connections, Secrets and Environment Variables +================================================== + +You can programmatically add Connections, Secrets and Environment Variables to your +airflow deployment using Helm chart. + + +Connections and Secrets +----------------------- +Under the ``secret`` and ``extraSecret`` sections of the ``values.yaml`` you can pass connection strings and secrets +into Airflow using Helm chart. To illustrate, lets create a yaml file to override values under these sections of the +``values.yaml`` file. + +.. code-block:: yaml + + secret: + - envName: "my-env" + secretName: "my-secret-name" + secretKey: "my-secret-key" + + + extraSecrets: + {{ .Release.Name }}-airflow-connections: + data: | + AIRFLOW_CONN_GCP: 'base64_encoded_gcp_conn_string' + AIRFLOW_CONN_AWS: 'base64_encoded_aws_conn_string' + stringData: | + AIRFLOW_CONN_OTHER: 'other_conn' + {{ .Release.Name }}-other-secret-name-suffix: | + data: | + + +Variables +--------- +Airflow supports Variables which enable users craft dynamic DAGs. You can set variables in Airflow in three ways - UI, +commandline, and within your DAG file. See:doc:`/howto/variable`.` for more. + +With Helm chart, you can also inject environment variables into airflow. For example, in a yaml file to +override values of interest in the ``env`` section of the ``values.yaml`` file. + +.. code-block:: yaml + + env: + - name: "AIRFLOW_VAR_KEY" + value: "value_1" + + - name: "AIRFLOW_VAR_ANOTHER_KEY" + value: "value_2" + + +You can also utilize the ``extraEnv`` and ``extraEnvFrom`` to set more possibly "templatable" values for your airflow +deployment. + +.. code-block:: yaml + + extraEnv: | + - name: AIRFLOW__CORE__LOAD_EXAMPLES + value: 'True' + + + extraEnvFrom: | + - secretRef: + name: '{{ .Release.Name }}-airflow-connections' + - configMapRef: + name: '{{ .Release.Name }}-airflow-variables' Review comment: ```suggestion extraEnvFrom: | - configMapRef: name: '{{ .Release.Name }}-airflow-variables' extraConfigMaps: '{{ .Release.Name }}-airflow-variables': data: | AIRFLOW_VAR_HELLO_MESSAGE: "Hi!" ``` Might as well be a complete working example. ########## File path: docs/helm-chart/adding-connections-and-variables.rst ########## @@ -0,0 +1,83 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + + +Adding Connections, Secrets and Environment Variables +================================================== + +You can programmatically add Connections, Secrets and Environment Variables to your +airflow deployment using Helm chart. + + +Connections and Secrets Review comment: ```suggestion Connections and Sensitive Environment Variables ``` Maybe? ########## File path: docs/helm-chart/adding-connections-and-variables.rst ########## @@ -0,0 +1,83 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + + +Adding Connections, Secrets and Environment Variables Review comment: ```suggestion Adding Connections, Variables and Environment Variables ``` I wouldn't consider `Secrets` to be core concept 🤷♂️ ########## File path: docs/helm-chart/adding-connections-and-variables.rst ########## @@ -0,0 +1,83 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + + +Adding Connections, Secrets and Environment Variables +================================================== + +You can programmatically add Connections, Secrets and Environment Variables to your +airflow deployment using Helm chart. + + +Connections and Secrets +----------------------- +Under the ``secret`` and ``extraSecret`` sections of the ``values.yaml`` you can pass connection strings and secrets +into Airflow using Helm chart. To illustrate, lets create a yaml file to override values under these sections of the +``values.yaml`` file. + +.. code-block:: yaml + + secret: + - envName: "my-env" + secretName: "my-secret-name" + secretKey: "my-secret-key" + + + extraSecrets: + {{ .Release.Name }}-airflow-connections: + data: | + AIRFLOW_CONN_GCP: 'base64_encoded_gcp_conn_string' + AIRFLOW_CONN_AWS: 'base64_encoded_aws_conn_string' + stringData: | + AIRFLOW_CONN_OTHER: 'other_conn' + {{ .Release.Name }}-other-secret-name-suffix: | + data: | + + +Variables +--------- +Airflow supports Variables which enable users craft dynamic DAGs. You can set variables in Airflow in three ways - UI, +commandline, and within your DAG file. See:doc:`/howto/variable`.` for more. + +With Helm chart, you can also inject environment variables into airflow. For example, in a yaml file to +override values of interest in the ``env`` section of the ``values.yaml`` file. + +.. code-block:: yaml + + env: + - name: "AIRFLOW_VAR_KEY" + value: "value_1" + Review comment: ```suggestion ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
