[
https://issues.apache.org/jira/browse/YARN-8250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16475100#comment-16475100
]
Haibo Chen commented on YARN-8250:
----------------------------------
[~leftnoteasy] Thanks for your comments.
I agree that we should avoid the CS vs FS issue if possible. As I have
mentioned, the rational is to do things that is suitable for Oversubscription
but do not break or de-stablize existing functionalities.
Can you elaborate on what do you mean by an issue we need to fix. I was
describing a behavior that is fine except for over-allocation. Today container
scheduler tries to launch opportunistic containers whenever there is a
container scheduling request, or whenever a container finishes. This is not an
issue today. But, in the case of over-allocation because of the fact that the
utilization metrics is stale, it is possible that we'd have the following case.
A few containers finishes, the container monitor checks the node utilization,
which is low, and then the container scheduler gets the container finish events
and aggressively tries to start opportunistic containers. Then later NM
realizes that opportunistic containers need to be preempted.
Not sure what can be done here to unify the two, as they fundamentally have
issues with the other one's approach. Hence, the proposal to have two
implementations.
{quote}) I'm not sure if we should give all the decisions to CGroups
{quote}
One key thing to note is that we want to ensure GUARANTEED containers are not
slowed down by OPPORTUNISTIC containers, so cgroup is always a requirement to
do over-allocation from day one to ensure isolation. Unless docker container
executor has similar mechanisms, it is hard to make over-allocation work
properly with docker without much downsides that render the feature unusable.
I am open to suggestions to make things simpler and more maintainable, but as
noted here, there is fundamental behavior changes. I'll try to take a look at
if there is more behaviors that we could extract into the base container
scheduler.
> Create another implementation of ContainerScheduler to support NM
> overallocation
> --------------------------------------------------------------------------------
>
> Key: YARN-8250
> URL: https://issues.apache.org/jira/browse/YARN-8250
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Haibo Chen
> Assignee: Haibo Chen
> Priority: Major
> Attachments: YARN-8250-YARN-1011.00.patch,
> YARN-8250-YARN-1011.01.patch, YARN-8250-YARN-1011.02.patch
>
>
> YARN-6675 adds NM over-allocation support by modifying the existing
> ContainerScheduler and providing a utilizationBased resource tracker.
> However, the implementation adds a lot of complexity to ContainerScheduler,
> and future tweak of over-allocation strategy based on how much containers
> have been launched is even more complicated.
> As such, this Jira proposes a new ContainerScheduler that always launch
> guaranteed containers immediately and queues opportunistic containers. It
> relies on a periodical check to launch opportunistic containers.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
