[
https://issues.apache.org/jira/browse/AIRFLOW-5700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
F D updated AIRFLOW-5700:
-------------------------
Description:
As a developer - I find myself struggling with the Airflow implementation of
launching pods within the KubeExecutor. Things are implemented in ways that are
difficult to get working with my kubernetes implementation, are difficult
because certain features of the Kubernetes Spec aren't implemented in a way
that is configurable, or features are forced to be implemented in a certain way
because of design decisions in the airflow wrappers around around the
kubernetes client. I believe supporting a YAML template would solve a lot of
problems and reduce the load on airflow maintainers for feature requests around
supporting more of the Kubernetes API
Dask has good Kubernetes support. They provide a hook to define a pod template
via YAML, OR to use defaults a thin abstraction via `make_pod_spec`
You can see an example here: https://kubernetes.dask.org/en/latest/#quickstart
and source here:
https://github.com/dask/dask-kubernetes/blob/master/dask_kubernetes/kubernetes.yaml#L35-L64
https://github.com/dask/dask-kubernetes/blob/master/dask_kubernetes/objects.py#L100-L163
https://github.com/dask/dask-kubernetes/blob/master/dask_kubernetes/core.py#L279-L291
https://github.com/dask/dask-kubernetes/blob/master/dask_kubernetes/core.py#L531-L566
was:
As a developer - I find myself struggling with the Airflow implementation of
launching pods within the KubeExecutor. Things are implemented in ways that are
difficult to get working with my kubernetes implementation, are difficult
because certain features of the Kubernetes Spec aren't implemented in a way
that is configurable, or features are forced to be implemented in a certain way
because of design decisions in the airflow wrappers around around the
kubernetes client.
Dask has good Kubernetes support. They provide a hook to define a pod template
via YAML, OR to use defaults a thin abstraction via `make_pod_spec`
You can see an example here: https://kubernetes.dask.org/en/latest/#quickstart
and source here:
https://github.com/dask/dask-kubernetes/blob/master/dask_kubernetes/kubernetes.yaml#L35-L64
https://github.com/dask/dask-kubernetes/blob/master/dask_kubernetes/objects.py#L100-L163
https://github.com/dask/dask-kubernetes/blob/master/dask_kubernetes/core.py#L279-L291
https://github.com/dask/dask-kubernetes/blob/master/dask_kubernetes/core.py#L531-L566
If Airflow implemented support for creating pods via a YAML template, new
feature requests and maintenance for the KubernetesExecutor would drop to very
little, and airflow users would have a lot of power and flexibility with what
pods look like in their system.
> KubeExecutor - Specify YAML template
> ------------------------------------
>
> Key: AIRFLOW-5700
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5700
> Project: Apache Airflow
> Issue Type: New Feature
> Components: executor-kubernetes
> Affects Versions: 1.10.5
> Reporter: F D
> Assignee: Daniel Imberman
> Priority: Critical
>
> As a developer - I find myself struggling with the Airflow implementation of
> launching pods within the KubeExecutor. Things are implemented in ways that
> are difficult to get working with my kubernetes implementation, are difficult
> because certain features of the Kubernetes Spec aren't implemented in a way
> that is configurable, or features are forced to be implemented in a certain
> way because of design decisions in the airflow wrappers around around the
> kubernetes client. I believe supporting a YAML template would solve a lot of
> problems and reduce the load on airflow maintainers for feature requests
> around supporting more of the Kubernetes API
> Dask has good Kubernetes support. They provide a hook to define a pod
> template via YAML, OR to use defaults a thin abstraction via `make_pod_spec`
> You can see an example here: https://kubernetes.dask.org/en/latest/#quickstart
> and source here:
> https://github.com/dask/dask-kubernetes/blob/master/dask_kubernetes/kubernetes.yaml#L35-L64
> https://github.com/dask/dask-kubernetes/blob/master/dask_kubernetes/objects.py#L100-L163
> https://github.com/dask/dask-kubernetes/blob/master/dask_kubernetes/core.py#L279-L291
> https://github.com/dask/dask-kubernetes/blob/master/dask_kubernetes/core.py#L531-L566
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
