[ 
https://issues.apache.org/jira/browse/AIRFLOW-4843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906258#comment-16906258
 ] 

ASF subversion and git services commented on AIRFLOW-4843:
----------------------------------------------------------

Commit 3e2a02751cf890b780bc26b40c7cee7f1f4e0bd9 in airflow's branch 
refs/heads/master from Akshesh Doshi
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=3e2a027 ]

[AIRFLOW-4843] Allow orchestration via Docker Swarm (SwarmOperator) (#5489)

* [AIRFLOW-4843] Allow orchestration via Docker Swarm (SwarmOperator)

Add support for running Docker containers via Docker Swarm
which allows the task to run on any machine (node) which
is a part of your Swarm cluster

More details: https://issues.apache.org/jira/browse/AIRFLOW-4843

Built with <3 at Agoda!

> Allow orchestration of tasks with Docker Swarm aka `SwarmOperator`
> ------------------------------------------------------------------
>
>                 Key: AIRFLOW-4843
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-4843
>             Project: Apache Airflow
>          Issue Type: New Feature
>          Components: operators
>    Affects Versions: 1.10.3
>            Reporter: Akshesh Doshi
>            Assignee: Akshesh Doshi
>            Priority: Major
>              Labels: Docker, docker, orchestration, swarm
>
> Currently, Airflow supports spawning Docker containers for running tasks via 
> the {color:#707070}_DockerOperator_{color} but these containers are run on 
> the same node as the scheduler.
> It would be helpful for our use-case to be able to spawn these tasks wherever 
> resources are available in our Docker Swarm cluster.
>  
> This can be achieved by creating a Docker swarm service, waiting for its run 
> and removing it after it has completed execution.
> This approach has been suggested/discussed at various places (and implemented 
> in Golang for Swarm-cronjob):
> [https://blog.alexellis.io/containers-on-swarm/]
> [https://forums.docker.com/t/running-one-off-commands-in-swarm-containers/42436/3]
> [https://gist.github.com/alexellis/e11321b8fbfc595c208ea3e74bf5e54b]
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Reply via email to