[
https://issues.apache.org/jira/browse/AIRFLOW-193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Imberman reassigned AIRFLOW-193:
---------------------------------------
Assignee: Daniel Imberman
> Allow a series of tasks to be executed on the same worker
> ---------------------------------------------------------
>
> Key: AIRFLOW-193
> URL: https://issues.apache.org/jira/browse/AIRFLOW-193
> Project: Apache Airflow
> Issue Type: New Feature
> Affects Versions: 1.7.1.2
> Reporter: Sergei Iakhnin
> Assignee: Daniel Imberman
> Priority: Major
>
> Currently the only way to limit the execution of a series of tasks to a
> single worker is via pools, however this is not a very convenient method when
> managing hundreds of workers.
> In the context of scientific workflows it is a common desire to be able to
> retrieve a (possibly large) sample from a data repository (or object store),
> then progressively elaborate it via a series of transformations, and finally
> deposit the result back. From a modelling perspective it makes sense to have
> the series of transformations each be encapsulated in a separate task. From
> practical considerations (performance, network bandwidth) it would be
> desirable to retrieve the sample to a single worker's local storage, where it
> would then be worked upon until completion. This, of course, requires the
> ability to have a slew of tasks to be bound to a particular worker.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
