[
https://issues.apache.org/jira/browse/YARN-8509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589226#comment-16589226
]
Eric Payne commented on YARN-8509:
----------------------------------
{code:title=UsersManager#computeUserLimit}
- Resource userLimitResource = Resources.max(resourceCalculator,
- partitionResource,
- Resources.divideAndCeil(resourceCalculator, resourceUsed,
- usersSummedByWeight),
- Resources.divideAndCeil(resourceCalculator,
- Resources.multiplyAndRoundDown(currentCapacity, getUserLimit()),
- 100));
+ Resource userLimitResource = Resources.multiplyAndRoundDown(queueCapacity,
+ getUserLimitFactor());
{code}
This is a drastic change that affects more than just preemption (the title of
this JIRA is "Total pending resource calculation in preemption should use
user-limit factor instead of minimum-user-limit-percent"). Forgive me if I
didn't understand that this JIRA is trying to change the way the capacity
scheduler calculates user limits.
[~leftnoteasy], I thought that the idea goal of the algorithm within
{{computeUserLimit}} is to slowly grow each queue once the queue is over it's
capacity so that resources can be assigned evenly. Are you okay with this
change?
> Total pending resource calculation in preemption should use user-limit factor
> instead of minimum-user-limit-percent
> -------------------------------------------------------------------------------------------------------------------
>
> Key: YARN-8509
> URL: https://issues.apache.org/jira/browse/YARN-8509
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
> Reporter: Zian Chen
> Assignee: Zian Chen
> Priority: Major
> Labels: capacityscheduler
> Attachments: YARN-8509.001.patch, YARN-8509.002.patch,
> YARN-8509.003.patch, YARN-8509.004.patch, YARN-8509.005.patch
>
>
> In LeafQueue#getTotalPendingResourcesConsideringUserLimit, we calculate total
> pending resource based on user-limit percent and user-limit factor which will
> cap pending resource for each user to the minimum of user-limit pending and
> actual pending. This will prevent queue from taking more pending resource to
> achieve queue balance after all queue satisfied with its ideal allocation.
>
> We need to change the logic to let queue pending can go beyond userlimit.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
