[
https://issues.apache.org/jira/browse/AIRFLOW-4008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Lum updated AIRFLOW-4008:
-------------------------------
Description:
This ticket is split off from AIRFLOW-3258. This addresses the problem of
having scheduler and worker pods with the same configuration. Manually
specifying environment variables in the deployment file and then again in the
{{airflow.cfg}} is error prone. Two options to fix this are:
# Add all the environment variables of the scheduler to the pods being brought
up. This could be set with a boolean flag in the kubernetes section of the
{{airflow.cfg}} which would to default to false, and have a name such as
copy_scheduler_enivronement. Any environment variables set in the
kubernetes_environment_variables or kuberenetes_secrets section, as well as
those that are hard coded such as AIRFLOW__CORE__EXECUTOR would overwrite those
present in the current environment when applied to the worker pod.
# Specify a ConfigMap that the worker pods {{envFrom}} in the {{airflow.cfg}}
that could act as a shared resource for the scheduler and worker pods.
was:
This ticket is split off from AIRFLOW-3258. This addresses that the problem of
having scheduler and worker pods with the same configuration. Manually
specifying environment variables in the deployment file and then again in the
{{airflow.cfg}} is error prone. Two options to fix this are:
# Add all the environment variables of the scheduler to the pods being brought
up. This could be set with a boolean flag in the kubernetes section of the
{{airflow.cfg}} which would to default to false, and have a name such as
copy_scheduler_enivronement. Any environment variables set in the
kubernetes_environment_variables or kuberenetes_secrets section, as well as
those that are hard coded such as AIRFLOW__CORE__EXECUTOR would overwrite those
present in the current environment when applied to the worker pod.
# Specify a ConfigMap that the worker pods {{envFrom}} in the {{airflow.cfg}}
that could act as a shared resource for the scheduler and worker pods.
> Add ability to set a ConfigMap to EnvFrom for pods brought up by the
> Kubernetes Executor
> ----------------------------------------------------------------------------------------
>
> Key: AIRFLOW-4008
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4008
> Project: Apache Airflow
> Issue Type: Improvement
> Components: kubernetes
> Reporter: David Lum
> Assignee: David Lum
> Priority: Minor
> Labels: Kubernetes, kubernetes
>
> This ticket is split off from AIRFLOW-3258. This addresses the problem of
> having scheduler and worker pods with the same configuration. Manually
> specifying environment variables in the deployment file and then again in the
> {{airflow.cfg}} is error prone. Two options to fix this are:
> # Add all the environment variables of the scheduler to the pods being
> brought up. This could be set with a boolean flag in the kubernetes section
> of the {{airflow.cfg}} which would to default to false, and have a name such
> as copy_scheduler_enivronement. Any environment variables set in the
> kubernetes_environment_variables or kuberenetes_secrets section, as well as
> those that are hard coded such as AIRFLOW__CORE__EXECUTOR would overwrite
> those present in the current environment when applied to the worker pod.
> # Specify a ConfigMap that the worker pods {{envFrom}} in the
> {{airflow.cfg}} that could act as a shared resource for the scheduler and
> worker pods.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
