[
https://issues.apache.org/jira/browse/YARN-4945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439148#comment-15439148
]
Eric Payne commented on YARN-4945:
----------------------------------
[~sunilg] and [~leftnoteasy], While I was writing the use cases, I thought
about the interaction between the inter-queue and intra-queue preemption
policies, and I want to get your thoughts. The following use case is not in the
doc, because I'm not sure how to handle it.
Consider this use case:
The cluster has 2 queues, both preemptable, both have configured max capacity
that can use the whole cluster.
||Queue Name||Queue Guaranteed Resources||Queue Max Resources||Queue
{{minimum-user-limit-percent}}||Queue Preemptable||
|root|200|200|N/A|N/A|
|{{QUEUE1}}|100|200|100%|Yes|
|{{QUEUE2}}|100|200|50%|Yes|
---
||Queue Name||User Name||App Name||Resources Used||Resources Guaranteed Per
User by
{{minimum-user-limit-percent}}||Pending Resources||
|{{QUEUE1}}|{{User1}}|{{App1}}|120|100|0|
|{{QUEUE2}}|{{User2}}|{{App2}}|80|50|0|
|{{QUEUE2}}|{{User3}}|{{App3}}|0|50|20|
# The inter-queue preemption policy sees that {{QUEUE2}} is underserved and is
asking for 20 resources, and {{QUEUE1}} is over-served by 20 resources, so it
preempts 20 resources from {{App1}}.
# The intra-queue preemption policy sees that {{User3}} is under its
{{minimum-user-limit-percent}} and is aksing for 20 resources, and {{User2}} is
over its {{minimum-user-limit-percent}}, so the intra-queue-preemption policy
preempts 20 resources from {{App2}}.
# The result of this scenario is that 20 resources are preempted when they
should not be.
The scenario I have laid out above assumes that intra-queue preemption did not
know about the 20 containers that are already preempted to fulfill the needs of
{{App3}} in {{QUEUE2}}. I think that the design doc tries to address this, and
assumes that the intra-queue preemption policy will be able to handle this use
case and will not preempt more containers when it is not necessary.
However, I am not so sure about that. In a more complicated scenario with
multiple over-served and multiple under-served queues, how will the intra-queue
preemption policy know that the containers that are already in the
{{selectedContainers}} list will be used to fulfill the needs of any specific
queue?
Please provide your thoughts.
> [Umbrella] Capacity Scheduler Preemption Within a queue
> -------------------------------------------------------
>
> Key: YARN-4945
> URL: https://issues.apache.org/jira/browse/YARN-4945
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Wangda Tan
> Attachments: Intra-Queue Preemption Use Cases.pdf,
> IntraQueuepreemption-CapacityScheduler (Design).pdf, YARN-2009-wip.patch
>
>
> This is umbrella ticket to track efforts of preemption within a queue to
> support features like:
> YARN-2009. YARN-2113. YARN-4781.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
