[
https://issues.apache.org/jira/browse/YARN-1488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13846343#comment-13846343
]
Arun C Murthy edited comment on YARN-1488 at 12/12/13 3:06 PM:
---------------------------------------------------------------
[~henryr] Good to see you around here.
bq. Would the recipient and delegated containers have to match the queues to
which their original resources were granted?
No, not at all.
The requirement, gleaned from discussions in YARN-1404 (and HDFS-4949, if you
squint hard *smile*) is that you'd want an external framework, call it Gamma,
which runs in a separate queue, call it GammaQ.
Now, a user Alice, belonging to queue Alpha, needs to use Gamma.
The key requirement is that Gamma would like to leverage YARN's workload
management capabilities (queues, SLAs etc.) rather than merely run under YARN
to leverage YARN's resource management.
Use-cases:
# Alice, belonging to queue Phi, running queries on Impala (resource: memory,
cpu, others in future)
# Bob, belonging to queue Alpha, caching data-sets in HDFS i.e. DataNodes
(resource: memory)
# Charlie, belonging to queue Beta, doing a bunch of I/O operations on
HBase/Accumulo (resource: cpu, iops).
If we all agree on the use cases; then it would be very critical to support
source and target containers belonging to different queues - that would be key
to allow these external frameworks to leverage YARN's workload management.
Does that make sense?
This would definitely require the NodeManager (and, potentially, the external
system i.e. impalad, datanode etc.) to maintain the resource-map so that they
can return the original source container to YARN for various reasons (finished
the task at hand, preemption to respect queue SLAs etc.)
We could, and should, allow the recipient service to decide how to manage the
resource map for itself (i.e. decouple that from how the NodeManager manages
the mapping) - this could be either a single cgroup (which the NodeManager has
to manage for the external framework anyway) or a hierarchy within.
Thoughts? Thanks.
was (Author: acmurthy):
[~henryr] Good to see you around here.
bq. Would the recipient and delegated containers have to match the queues to
which their original resources were granted?
No, not at all.
The requirement, gleaned from discussions in YARN-1404 (and HDFS-4949, if you
squint hard *smile*) is that you'd want an external framework, call it Gamma,
which runs in a separate queue, call it GammaQ.
Now, a user Alice, belonging to queue Phi, needs to use Gamma.
The key requirement is that Gamma would like to leverage YARN's workload
management capabilities (queues, SLAs etc.) rather than merely run under YARN
to leverage YARN's resource management.
Use-cases:
# Alice running queries on Impala (resource: memory, cpu, others in future)
# Bob caching data-sets in HDFS i.e. DataNodes (resource: memory)
# Charlie doing a bunch of I/O operations on HBase/Accumulo (resource: cpu,
iops).
If we all agree on the use cases; then it would be very critical to support
source and target containers belonging to different queues - that would be key
to allow these external frameworks to leverage YARN's workload management.
Does that make sense?
This would definitely require the NodeManager (and, potentially, the external
system i.e. impalad, datanode etc.) to maintain the resource-map so that they
can return the original source container to YARN for various reasons (finished
the task at hand, preemption to respect queue SLAs etc.)
We could, and should, allow the recipient service to decide how to manage the
resource map for itself (i.e. decouple that from how the NodeManager manages
the mapping) - this could be either a single cgroup (which the NodeManager has
to manage for the external framework anyway) or a hierarchy within.
Thoughts? Thanks.
> Allow containers to delegate resources to another container
> -----------------------------------------------------------
>
> Key: YARN-1488
> URL: https://issues.apache.org/jira/browse/YARN-1488
> Project: Hadoop YARN
> Issue Type: New Feature
> Reporter: Arun C Murthy
>
> We should allow containers to delegate resources to another container. This
> would allow external frameworks to share not just YARN's resource-management
> capabilities but also it's workload-management capabilities.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
