[
https://issues.apache.org/jira/browse/YARN-1488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13846001#comment-13846001
]
Henry Robinson commented on YARN-1488:
--------------------------------------
Arun, thanks for filing this. I have a few questions so that I can understand
the proposal more concretely (I'm viewing this in the context of YARN-1404;
i.e. a server process that wants to obtain resources for workers that may not
have a 1-to-1 relationship with processes):
Would the recipient and delegated containers have to match the queues to which
their original resources were granted? If the target is the server process, and
the source is a set of resources granted for a single worker, the queues would
likely be different.
* If the queues do not have to match, then presumably the target container is
the server's. Would the server process now have to track all resources
delegated to it? I'm thinking that pre-empting a delegated container would
require the server process to ensure that it no longer assigns resources to the
relevant worker. This does not affect YARN's correctness, since the resources
will be revoked no matter what, but otherwise increases the likelihood that
YARN's understanding of the resource map of the cluster is inaccurate, which is
not good for anyone.
* If the queues *must* match, then presumably the target container is supposed
to be for the worker, not the server (because server and worker occupy
different queues). But if that's the case, the target container must already
exist, which makes it seem like YARN would require a process to be running in
it, getting us back to the start with our requirements for Impala, which is
that we'd like to have one container per query but we use that container by
assigning threads to it dynamically, rather than whole processes.
I like the delegation idea as a mechanism to coalesce resource requests into
one. But I don't yet understand how this allows us to maintain one
cgroup-per-worker without a dedicated worker process, unless the server process
creates a hierarchy of cgroups underneath it, one for each worker, and
physically delegates resources that way. This (or an alternative approach where
all workers run inside the same monolithic server cgroup, and the server
schedules them in user-land itself) places some pretty hefty requirements on
the framework to avoid misusing the resources that YARN grants it.
> Allow containers to delegate resources to another container
> -----------------------------------------------------------
>
> Key: YARN-1488
> URL: https://issues.apache.org/jira/browse/YARN-1488
> Project: Hadoop YARN
> Issue Type: New Feature
> Reporter: Arun C Murthy
>
> We should allow containers to delegate resources to another container. This
> would allow external frameworks to share not just YARN's resource-management
> capabilities but also it's workload-management capabilities.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
