[
https://issues.apache.org/jira/browse/YARN-8250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16476292#comment-16476292
]
Arun Suresh commented on YARN-8250:
-----------------------------------
[~haibochen], with regard to this:
{quote}
avoid aggressive OPPORTUNISTIC container launching. One thing to note is that
in case of over-allocation, we'd rely on the resource utilization metrics to
decide how much resources we can to launch OPPORTUNISTIC containers. The
resource utilization metrics in NM is unfortunately only updated every few
seconds. This can be problematic in that NM could end up with launching too
many OPPORTUNISTIC containers before the metric is updated. The current default
container scheduler launches containers aggressively, which could cause
containers to be launched and killed shortly after. The new container
scheduler only schedule OPPORTUNISTIC containers once whenever the utilization
metric is updated.
{quote}
IMHO, the whole point of opportunistic containers is to start them as fast as
possible. Even if the resource utilization is updated once every few seconds (I
am guessing this should be configurable - and therefore can be reduced) It
shouldnt matter if opportunistic containers are launched aggressively. The
scheduler cannot launch more than the available resources anyway. I agree there
can be churn - but think of it this way, these O containers were scheduled on
the node with the assumption that they have a higher likelyhood of being
preempted anyway - so why not just start them and let the container get
pre-empted when resources are low. Furthermore, once pause and resume is turned
on, there will technically be no lost work either.
> Create another implementation of ContainerScheduler to support NM
> overallocation
> --------------------------------------------------------------------------------
>
> Key: YARN-8250
> URL: https://issues.apache.org/jira/browse/YARN-8250
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Haibo Chen
> Assignee: Haibo Chen
> Priority: Major
> Attachments: YARN-8250-YARN-1011.00.patch,
> YARN-8250-YARN-1011.01.patch, YARN-8250-YARN-1011.02.patch
>
>
> YARN-6675 adds NM over-allocation support by modifying the existing
> ContainerScheduler and providing a utilizationBased resource tracker.
> However, the implementation adds a lot of complexity to ContainerScheduler,
> and future tweak of over-allocation strategy based on how much containers
> have been launched is even more complicated.
> As such, this Jira proposes a new ContainerScheduler that always launch
> guaranteed containers immediately and queues opportunistic containers. It
> relies on a periodical check to launch opportunistic containers.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
