dimberman opened a new issue #7920: Allow a series of tasks to be executed on the same worker URL: https://github.com/apache/airflow/issues/7920 **Description** Currently the only way to limit the execution of a series of tasks to a single worker is via pools, however this is not a very convenient method when managing hundreds of workers. In the context of scientific workflows it is a common desire to be able to retrieve a (possibly large) sample from a data repository (or object store), then progressively elaborate it via a series of transformations, and finally deposit the result back. From a modelling perspective it makes sense to have the series of transformations each be encapsulated in a separate task. From practical considerations (performance, network bandwidth) it would be desirable to retrieve the sample to a single worker's local storage, where it would then be worked upon until completion. This, of course, requires the ability to have a slew of tasks to be bound to a particular worker. **Use case / motivation** **Related Issues** Moved here from https://issues.apache.org/jira/browse/AIRFLOW-193
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
