[
https://issues.apache.org/jira/browse/YARN-10821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17363427#comment-17363427
]
Andras Gyori commented on YARN-10821:
-------------------------------------
The full call chain is the following:
# TempQueuePerPartition#offer -> this method calculates the accepted resources
from preemption based on pending
# TempQueuePerPartition.pending
# LeafQueue#getTotalPendingResourcesConsideringUserLimit
# UsersManager#computeUserLimit
What we have also observed, is that by turning of minimum-user-limit-percent
(setting it to 100), there was no issue. But when we set MULP to say 50
percent, the queue have only been granted half of its effective capacity, even
though the other queue was using 600% of its effective capacity and preemption
should have kicked in.
> User limit is not calculated as per definition for preemption
> -------------------------------------------------------------
>
> Key: YARN-10821
> URL: https://issues.apache.org/jira/browse/YARN-10821
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
> Reporter: Andras Gyori
> Assignee: Andras Gyori
> Priority: Major
> Attachments: YARN-10821.001.patch
>
>
> Minimum user limit percent (MULP) is a soft limit by definition. Preemption
> uses pending resources to determine the resources needed by a queue, which is
> calculated in LeafQueue#getTotalPendingResourcesConsideringUserLimit. This
> method involves headroom calculated by UsersManager#computeUserLimit.
> However, the pending resources for preemption are limited in an unexpected
> fashion.
> * In LeafQueue#getUserAMResourceLimitPerPartition an effective userLimit is
> calculated first:
> {code:java}
> float effectiveUserLimit = Math.max(usersManager.getUserLimit() / 100.0f,
> 1.0f / Math.max(getAbstractUsersManager().getNumActiveUsers(), 1));
> {code}
> * In UsersManager#computeUserLimit the userLimit is calculated as is
> (currentCapacity * userLimit)
> {code:java}
> Resource userLimitResource = Resources.max(resourceCalculator,
> partitionResource,
> Resources.divideAndCeil(resourceCalculator, resourceUsed,
> usersSummedByWeight),
> Resources.divideAndCeil(resourceCalculator,
> Resources.multiplyAndRoundDown(currentCapacity, getUserLimit()),
> 100));
> {code}
> The fewer users occupying the queue, the more prevalent and outstanding this
> effect will be in preemption.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
