[
https://issues.apache.org/jira/browse/AIRFLOW-4843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor updated AIRFLOW-4843:
---------------------------------------
Fix Version/s: (was: 1.10.6)
1.10.7
> Allow orchestration of tasks with Docker Swarm aka `SwarmOperator`
> ------------------------------------------------------------------
>
> Key: AIRFLOW-4843
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4843
> Project: Apache Airflow
> Issue Type: New Feature
> Components: operators
> Affects Versions: 1.10.3
> Reporter: Akshesh Doshi
> Assignee: Akshesh Doshi
> Priority: Major
> Labels: Docker, docker, orchestration, swarm
> Fix For: 1.10.7
>
>
> Currently, Airflow supports spawning Docker containers for running tasks via
> the {color:#707070}_DockerOperator_{color} but these containers are run on
> the same node as the scheduler.
> It would be helpful for our use-case to be able to spawn these tasks wherever
> resources are available in our Docker Swarm cluster.
>
> This can be achieved by creating a Docker swarm service, waiting for its run
> and removing it after it has completed execution.
> This approach has been suggested/discussed at various places (and implemented
> in Golang for Swarm-cronjob):
> [https://blog.alexellis.io/containers-on-swarm/]
> [https://forums.docker.com/t/running-one-off-commands-in-swarm-containers/42436/3]
> [https://gist.github.com/alexellis/e11321b8fbfc595c208ea3e74bf5e54b]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
