James Davidheiser created AIRFLOW-2317:
------------------------------------------
Summary: Support for multiple resource pools
Key: AIRFLOW-2317
URL: https://issues.apache.org/jira/browse/AIRFLOW-2317
Project: Apache Airflow
Issue Type: Wish
Components: pools
Reporter: James Davidheiser
We are migrating to Airflow from Luigi, where we have the capability to require
multiple pools, and multiple "units" of a pool, for a given task. This is very
useful for a variety of use cases, but the two core examples are:
* If a task accessing a data store is extremely resource-intensive, we might
define a pool size of 10 resources, but use 5 of those in a single task to
denote the extra load. When smaller tasks are using the same data store, it's
completely fine to allow more of them to run at once.
If a task is connecting to two different data stores, to transfer data between
them, we might want to require a resource for each data store, so we can limit
concurrency on both simultaneously.
I know there are a lot of other issues related to capacity scheduling, which
are tracked more broadly in https://issues.apache.org/jira/browse/AIRFLOW-72,
and using more than one pool slot was suggested in
https://issues.apache.org/jira/browse/AIRFLOW-1467, so this seems like it could
be a useful thing to consider as part of larger efforts around managing
capacity.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
