[
https://issues.apache.org/jira/browse/YARN-8509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16576342#comment-16576342
]
Eric Payne edited comment on YARN-8509 at 8/10/18 6:12 PM:
-----------------------------------------------------------
[~Zian Chen], can I please get a couple of clarifications?
{quote}
total_pending(partition,queue) = min {Q_max(partition) - Q_used(partition), Σ
(min \{
User.ulf\(partition\) - User.used\(partition\), User.pending\(partition\}\)\}
{quote}
1) In the above pseudo-code, what is being summed by the summation?
2) In the above example, queue-a is the only one that's underserved, so the the
first round of preemption should actually preempt 6G from queues b and c. The
amount preempted from each queue depends on the age of the containers, but you
could end up with something like queue-b consuming 40G and pending 30G and
queue-c consuming 44G and pending 36G before the second round of preemption, at
which point queue-a would be satisfied and only queues b and c have pending
resource requests. Since this issue is meant to address the balancing of queues
that are over their capacity, I don't understand why queue-a is involved in the
above use case. Can you provide a simpler example that only involves the
balancing of over-served queues?
was (Author: eepayne):
[~Zian Chen], can I please get a couple of clarifications?
{quote}total_pending(partition,queue) = min {Q_max(partition) -
Q_used(partition), Σ (min
Unknown macro: \{User.ulf(partition) - User.used(partition),
User.pending(partition})}{quote}
1) In the above pseudo-code, what is being summed by the summation?
2) In the above example, queue-a is the only one that's underserved, so the the
first round of preemption should actually preempt 6G from queues b and c. The
amount preempted from each queue depends on the age of the containers, but you
could end up with something like queue-b consuming 40G and pending 30G and
queue-c consuming 44G and pending 36G before the second round of preemption, at
which point queue-a would be satisfied and only queues b and c have pending
resource requests. Since this issue is meant to address the balancing of queues
that are over their capacity, I don't understand why queue-a is involved in the
above use case. Can you provide a simpler example that only involves the
balancing of over-served queues?
> Total pending resource calculation in preemption should use user-limit factor
> instead of minimum-user-limit-percent
> -------------------------------------------------------------------------------------------------------------------
>
> Key: YARN-8509
> URL: https://issues.apache.org/jira/browse/YARN-8509
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
> Reporter: Zian Chen
> Assignee: Zian Chen
> Priority: Major
> Attachments: YARN-8509.001.patch, YARN-8509.002.patch,
> YARN-8509.003.patch
>
>
> In LeafQueue#getTotalPendingResourcesConsideringUserLimit, we calculate total
> pending resource based on user-limit percent and user-limit factor which will
> cap pending resource for each user to the minimum of user-limit pending and
> actual pending. This will prevent queue from taking more pending resource to
> achieve queue balance after all queue satisfied with its ideal allocation.
>
> We need to change the logic to let queue pending can go beyond userlimit.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
