[
https://issues.apache.org/jira/browse/YARN-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15157248#comment-15157248
]
Karthik Kambatla commented on YARN-1011:
----------------------------------------
bq. Could you clarify 2.2? What would trigger this if not nodeUpdate()?
We could just periodically go through all the nodes and allocate containers.
This is very similar to continuous/asynchronous scheduling in Fair/Capacity
schedulers.
bq. 2.3. Whats the reasoning behind this? Over-allocating a node seems to be a
local decision based on the nodes expected and actual utilization. So I would
expect the logic to be something similar to 1) Node is already 100% allocated
2) Actual utilization is < 80% 3) Over-allocate to bring actual utilization
~=80%.
There is a node config that determines if the node allows oversubscription and
by how much. That said, the RM still has to decide when/where to allocate
opportunistic containers. When the overall cluster utilization is low, it is
highly likely the RM would find a guaranteed container soon after it allocates
an opportunistic container for a ResourceRequest. By waiting for this
utilization to be over a threshold, we are avoiding having to promote
containers right after allocating them. This shouldn't be a problem in
practice, because we expect over-allocation to help improve the utilization on
a fully-allocated cluster.
{quote}
3. What is the AM/RM interaction in this promotion?
3.2. Not clear what is actually happening here? Will new container be allocated
and the opportunistic container allowed to continue till is exits or is
preempted?
{quote}
We don't know yet. :)
Promoting a container on the same node is fairly straight-forward: the node
just promotes the container and the AM can be informed that a running container
has been promoted should it want to differentiate between opportunistic and
guaranteed containers.
I am not actively thinking about promotion across nodes. Given the additional
complexity, I feel we should see some numbers before going further. And, the
rest of the work is required anyway.
> [Umbrella] Schedule containers based on utilization of currently allocated
> containers
> -------------------------------------------------------------------------------------
>
> Key: YARN-1011
> URL: https://issues.apache.org/jira/browse/YARN-1011
> Project: Hadoop YARN
> Issue Type: New Feature
> Reporter: Arun C Murthy
> Attachments: yarn-1011-design-v0.pdf, yarn-1011-design-v1.pdf,
> yarn-1011-design-v2.pdf
>
>
> Currently RM allocates containers and assumes resources allocated are
> utilized.
> RM can, and should, get to a point where it measures utilization of allocated
> containers and, if appropriate, allocate more (speculative?) containers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
