[ 
https://issues.apache.org/jira/browse/YARN-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14067542#comment-14067542
 ] 

Wangda Tan commented on YARN-2008:
----------------------------------

Hi [~cwelch],
Thanks for working on this patch. However, I've thought about this for a while, 
I'm wondering if we should change this behavior.
With preemption, we don't need consider used capacity of sibling or sibling of 
parents. Preemption policy will take care of over used queues. In addition, 
even if we have preemption disabled, the headroom should not be changed as well 
(see next).
If we define headroom as "maximum capacity of an application can get", the 
formula headroom = min((userLimit, queue-max-cap) - consumed) should be 
correct. But if we define headroom as "maximum *guaranteed* capacity of an 
application can get", the formula should be changed to headroom = 
min((userLimit, queue-max-cap, queue-guaranteed-cap) - consumed).

Does this make sense to you? Please let me know if you have any comments.

Thanks,
Wangda

> CapacityScheduler may report incorrect queueMaxCap if there is hierarchy 
> queue structure 
> -----------------------------------------------------------------------------------------
>
>                 Key: YARN-2008
>                 URL: https://issues.apache.org/jira/browse/YARN-2008
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>    Affects Versions: 2.3.0
>            Reporter: Chen He
>            Assignee: Chen He
>         Attachments: YARN-2008.1.patch, YARN-2008.2.patch
>
>
> If there are two queues, both allowed to use 100% of the actual resources in 
> the cluster. Q1 and Q2 currently use 50% of actual cluster's resources and 
> there is not actual space available. If we use current method to get 
> headroom, CapacityScheduler thinks there are still available resources for 
> users in Q1 but they have been used by Q2. 
> If the CapacityScheduelr has a hierarchy queue structure, it may report 
> incorrect queueMaxCap. Here is a example
>                              ||                    ||rootQueue||     ||
> |  |                               /                               |          
>           \                     |
> |      L1ParentQueue1                      |          |            
> L1ParentQueue2    |
> |  (allowed to use up 80% of its parent)    |  |         (allowed to use 20% 
> in minimum of its parent)|
> |                    /   |     \ |                            |  
> |  L2LeafQueue1 |    L2LeafQueue2 |  |     
> |(50% of its parent) |  (50% of its parent in minimum) |   |
> When we calculate headroom of a user in L2LeafQueue2, current method will 
> think L2LeafQueue2 can use 40% (80%*50%) of actual rootQueue resources. 
> However, without checking L1ParentQueue1, we are not sure. It is possible 
> that L1ParentQueue2 have used 40% of rootQueue resources right now. Actually, 
> L2LeafQueue2 can only use 30% (60%*50%). 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Reply via email to