[ https://issues.apache.org/jira/browse/AIRFLOW-4843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906258#comment-16906258 ]
ASF subversion and git services commented on AIRFLOW-4843: ---------------------------------------------------------- Commit 3e2a02751cf890b780bc26b40c7cee7f1f4e0bd9 in airflow's branch refs/heads/master from Akshesh Doshi [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=3e2a027 ] [AIRFLOW-4843] Allow orchestration via Docker Swarm (SwarmOperator) (#5489) * [AIRFLOW-4843] Allow orchestration via Docker Swarm (SwarmOperator) Add support for running Docker containers via Docker Swarm which allows the task to run on any machine (node) which is a part of your Swarm cluster More details: https://issues.apache.org/jira/browse/AIRFLOW-4843 Built with <3 at Agoda! > Allow orchestration of tasks with Docker Swarm aka `SwarmOperator` > ------------------------------------------------------------------ > > Key: AIRFLOW-4843 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4843 > Project: Apache Airflow > Issue Type: New Feature > Components: operators > Affects Versions: 1.10.3 > Reporter: Akshesh Doshi > Assignee: Akshesh Doshi > Priority: Major > Labels: Docker, docker, orchestration, swarm > > Currently, Airflow supports spawning Docker containers for running tasks via > the {color:#707070}_DockerOperator_{color} but these containers are run on > the same node as the scheduler. > It would be helpful for our use-case to be able to spawn these tasks wherever > resources are available in our Docker Swarm cluster. > > This can be achieved by creating a Docker swarm service, waiting for its run > and removing it after it has completed execution. > This approach has been suggested/discussed at various places (and implemented > in Golang for Swarm-cronjob): > [https://blog.alexellis.io/containers-on-swarm/] > [https://forums.docker.com/t/running-one-off-commands-in-swarm-containers/42436/3] > [https://gist.github.com/alexellis/e11321b8fbfc595c208ea3e74bf5e54b] > -- This message was sent by Atlassian JIRA (v7.6.14#76016)