[ 
https://issues.apache.org/jira/browse/YARN-8509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16576342#comment-16576342
 ] 

Eric Payne commented on YARN-8509:
----------------------------------

[~Zian Chen], can I please get a couple of clarifications?
{quote}total_pending(partition,queue) = min {Q_max(partition) - 
Q_used(partition), Σ (min
Unknown macro: \{User.ulf(partition) - User.used(partition), 
User.pending(partition})}{quote}
1) In the above pseudo-code, what is being summed by the summation?

2) In the above example, queue-a is the only one that's underserved, so the the 
first round of preemption should actually preempt 6G from queues b and c. The 
amount preempted from each queue depends on the age of the containers, but you 
could end up with something like queue-b consuming 40G and pending 30G and 
queue-c consuming 44G and pending 36G before the second round of preemption, at 
which point queue-a would be satisfied and only queues b and c have pending 
resource requests. Since this issue is meant to address the balancing of queues 
that are over their capacity, I don't understand why queue-a is involved in the 
above use case. Can you provide a simpler example that only involves the 
balancing of over-served queues?

> Total pending resource calculation in preemption should use user-limit factor 
> instead of minimum-user-limit-percent
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-8509
>                 URL: https://issues.apache.org/jira/browse/YARN-8509
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: yarn
>            Reporter: Zian Chen
>            Assignee: Zian Chen
>            Priority: Major
>         Attachments: YARN-8509.001.patch, YARN-8509.002.patch, 
> YARN-8509.003.patch
>
>
> In LeafQueue#getTotalPendingResourcesConsideringUserLimit, we calculate total 
> pending resource based on user-limit percent and user-limit factor which will 
> cap pending resource for each user to the minimum of user-limit pending and 
> actual pending. This will prevent queue from taking more pending resource to 
> achieve queue balance after all queue satisfied with its ideal allocation.
>   
>  We need to change the logic to let queue pending can go beyond userlimit.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

Reply via email to