[
https://issues.apache.org/jira/browse/YARN-8178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16449842#comment-16449842
]
Weiwei Yang commented on YARN-8178:
-----------------------------------
Hi [~Zian Chen]
This is a very interesting idea. The design doc looks very nice. Here are some
questions I could not figure out via reading the doc,
1. Highlevel
Can user decide if their app can be demoted or not?
2. How to select which apps/containers to demote?
Could you please elaborate the 2nd paragraph in \{{Make more aggressive
preemption interval}}, if possible, with an example? I am curious how it works
together with current preemption logic. When it needs to demote containers, how
to select apps and containers? A policy is pretty important, because you don't
want to end up with some inefficient demotions, such as two many apps gets
affected, apps get starved etc.
3. O container lifecycle
How to manage the life cycle of O containers, as you mentioned in the doc, it
will continue to run unless RM kills it or gets preempted. Bad case scenario,
what if a O container continues to consume nodes resource, will NM be able to
kill it when it exceeds some threshold?
Looking forward for more updates!
Thanks
> [Umbrella] Resource Over-commitment Based on Opportunistic Container
> Preemption
> -------------------------------------------------------------------------------
>
> Key: YARN-8178
> URL: https://issues.apache.org/jira/browse/YARN-8178
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: capacity scheduler
> Reporter: Zian Chen
> Priority: Major
> Attachments: Resource Over-commitment Based on Opportunistic
> Container Preemption.pdf
>
>
> We want to provide an opportunistic container-based solution to achieve more
> aggressive preemption with shorter preemption monitoring interval.
> Instead of allowing applications to allocate resources with a mix of
> guaranteed and opportunistic containers, we allow newly submitted
> applications to only contain guaranteed containers. Meanwhile, we change the
> preemption logic to, instead of killing containers, demote guaranteed
> containers into opportunistic ones, so that when there are new applications
> submitted, we can ensure that these containers can be launched by preempting
> opportunistic containers.
> This approach is related to YARN-1011 but achieves over-commitment in a
> different way. However, we rely on opportunistic container part implemented
> in YARN-1011 to make our design work well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
