[
https://issues.apache.org/jira/browse/AIRFLOW-2317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor resolved AIRFLOW-2317.
----------------------------------------
Resolution: Duplicate
> Support for multiple resource pools
> -----------------------------------
>
> Key: AIRFLOW-2317
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2317
> Project: Apache Airflow
> Issue Type: Wish
> Components: scheduler
> Reporter: James Davidheiser
> Priority: Minor
> Labels: pool
>
> We are migrating to Airflow from Luigi, where we have the capability to
> require multiple pools, and multiple "units" of a pool, for a given task.
> This is very useful for a variety of use cases, but the two core examples are:
> * If a task accessing a data store is extremely resource-intensive, we might
> define a pool size of 10 resources, but use 5 of those in a single task to
> denote the extra load. When smaller tasks are using the same data store,
> it's completely fine to allow more of them to run at once.
> If a task is connecting to two different data stores, to transfer data
> between them, we might want to require a resource for each data store, so we
> can limit concurrency on both simultaneously.
>
>
> I know there are a lot of other issues related to capacity scheduling, which
> are tracked more broadly in https://issues.apache.org/jira/browse/AIRFLOW-72,
> and using more than one pool slot was suggested in
> https://issues.apache.org/jira/browse/AIRFLOW-1467, so this seems like it
> could be a useful thing to consider as part of larger efforts around managing
> capacity.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
