[
https://issues.apache.org/jira/browse/AIRFLOW-2745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eamon Keane updated AIRFLOW-2745:
---------------------------------
Description:
When deploying airflow on kubernetes and using LocalExecutor, currently the
Kubernetes Pod Operator relies on a kubeconfig file being mounted on the
scheduler and has no awareness of being inside a cluster.
There is no option to mount a kubeconfig file on a KubernetesExecutor worker as
the KubernetesExecutor instead launches Kubenetes Pod Operator pods using the
mounted RBAC account.
For users switching between the KubernetesExecutor and LocalExecutor in a helm
chart (for example by using --set core.executor=LocalExecutor), an additional
kubeconfig secret has to be managed and mounted on the scheduler if they want
to debug a dag which uses the Kubernetes Pod Operator, while it could instead
use the RBAC account.
An example where switching between local and kubernetes executor was useful was
to discover that the reason a dag worked on Local but not on Kubernetes
Executor was because the fernet key was not specified as an environment
variable in the worker definition.
The suggested improvement would be to use the mounted RBAC account on the
scheduler pod to launch pods if in a kubernetes environment, removing the need
for a kubeconfig.
was:
When deploying airflow on kubernetes and using LocalExecutor, currently the
Kubernetes Pod Operator relies on a kubeconfig file being mounted on the
scheduler and has no awareness of being inside a cluster.
There is no option to mount a kubeconfig file on a KubernetesExecutor worker as
the KubernetesExecutor instead launches Kubenetes Pod Operator pods using the
mounted RBAC account.
For users switching between the KubernetesExecutor and LocalExecutor in a helm
chart (for example by using --set core.executor=LocalExecutor), an additional
kubeconfig secret has to be managed and mounted on scheduler if they want to
debug a dag which uses the Kubernetes Pod Operator, while it could instead use
the RBAC account.
An example where switching between local and kubernetes executor was useful was
to discover that the reason a dag worked on Local but not on Kubernetes
Executor was because the fernet key was not specified as an environment
variable in the worker definition.
The suggested improvement would be to use the mounted RBAC account on the
scheduler pod to launch pods if in a kubernetes environment, removing the need
for a kubeconfig.
> Use k8s service account for Kube Pod Operator if in Cluster and using
> LocalExecutor
> -----------------------------------------------------------------------------------
>
> Key: AIRFLOW-2745
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2745
> Project: Apache Airflow
> Issue Type: Improvement
> Components: operators
> Affects Versions: 2.0.0
> Reporter: Eamon Keane
> Assignee: Daniel Imberman
> Priority: Minor
> Labels: features
>
> When deploying airflow on kubernetes and using LocalExecutor, currently the
> Kubernetes Pod Operator relies on a kubeconfig file being mounted on the
> scheduler and has no awareness of being inside a cluster.
> There is no option to mount a kubeconfig file on a KubernetesExecutor worker
> as the KubernetesExecutor instead launches Kubenetes Pod Operator pods using
> the mounted RBAC account.
> For users switching between the KubernetesExecutor and LocalExecutor in a
> helm chart (for example by using --set core.executor=LocalExecutor), an
> additional kubeconfig secret has to be managed and mounted on the scheduler
> if they want to debug a dag which uses the Kubernetes Pod Operator, while it
> could instead use the RBAC account.
> An example where switching between local and kubernetes executor was useful
> was to discover that the reason a dag worked on Local but not on Kubernetes
> Executor was because the fernet key was not specified as an environment
> variable in the worker definition.
> The suggested improvement would be to use the mounted RBAC account on the
> scheduler pod to launch pods if in a kubernetes environment, removing the
> need for a kubeconfig.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
