[ https://issues.apache.org/jira/browse/AIRFLOW-5700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
F D updated AIRFLOW-5700: ------------------------- Description: As a developer - I find myself struggling with the Airflow implementation of launching pods within the KubeExecutor. Things are implemented in ways that are difficult to get working with my kubernetes implementation, are difficult because certain features of the Kubernetes Spec aren't implemented in a way that is configurable, or features are forced to be implemented in a certain way because of design decisions in the airflow wrappers around around the kubernetes client. Dask has really good Kubernetes support. They provide a hook to define a pod template via YAML, OR to use defaults a thin abstraction via `make_pod_spec` You can see an example here: https://kubernetes.dask.org/en/latest/#quickstart and source here: https://github.com/dask/dask-kubernetes/blob/master/dask_kubernetes/kubernetes.yaml#L35-L64 https://github.com/dask/dask-kubernetes/blob/master/dask_kubernetes/objects.py#L100-L163 https://github.com/dask/dask-kubernetes/blob/master/dask_kubernetes/core.py#L279-L291 If Airflow implemented support for creating pods via a YAML template, new feature requests and maintenance for the KubernetesExecutor would drop to very little, and airflow users would have a lot of power and flexibility with what pods look like in their system. was: As a developer - I find myself struggling with the Airflow implementation of launching pods within the KubeExecutor. Things are implemented in ways that don't work with my kubernetes implementation, are unnecessarily difficult because certain features of the Kubernetes Spec aren't implemented in a way I can configure, or features are forced to be implemented in a certain way because of historical design decisions by the airflow team. Dask has really good Kubernetes support. They provide a hook to define a pod template via YAML, OR to use defaults. You can see an example here: https://kubernetes.dask.org/en/latest/#quickstart If Airflow implemented support for creating pods via a YAML template, new feature requests and maintenance for the KubernetesExecutor would drop to very little, and airflow users would have a lot of power and flexibility with what pods look like in their system. > KubeExecutor - Specify YAML template > ------------------------------------ > > Key: AIRFLOW-5700 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5700 > Project: Apache Airflow > Issue Type: New Feature > Components: executor-kubernetes > Affects Versions: 1.10.5 > Reporter: F D > Assignee: Daniel Imberman > Priority: Critical > > As a developer - I find myself struggling with the Airflow implementation of > launching pods within the KubeExecutor. Things are implemented in ways that > are difficult to get working with my kubernetes implementation, are difficult > because certain features of the Kubernetes Spec aren't implemented in a way > that is configurable, or features are forced to be implemented in a certain > way because of design decisions in the airflow wrappers around around the > kubernetes client. > Dask has really good Kubernetes support. They provide a hook to define a pod > template via YAML, OR to use defaults a thin abstraction via `make_pod_spec` > You can see an example here: https://kubernetes.dask.org/en/latest/#quickstart > and source here: > https://github.com/dask/dask-kubernetes/blob/master/dask_kubernetes/kubernetes.yaml#L35-L64 > https://github.com/dask/dask-kubernetes/blob/master/dask_kubernetes/objects.py#L100-L163 > https://github.com/dask/dask-kubernetes/blob/master/dask_kubernetes/core.py#L279-L291 > If Airflow implemented support for creating pods via a YAML template, new > feature requests and maintenance for the KubernetesExecutor would drop to > very little, and airflow users would have a lot of power and flexibility with > what pods look like in their system. -- This message was sent by Atlassian Jira (v8.3.4#803005)