[
https://issues.apache.org/jira/browse/YARN-10821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17376054#comment-17376054
]
Eric Payne commented on YARN-10821:
-----------------------------------
{quote}
{noformat}
max ((total resources consumed by active users) / (number of active users)),
((totalQueueResourcesAvailableToUsers) * min-user-limit-pct))
{noformat}
bq. UserA is not using any resource yet, and no one else is using QueueA. Is
not "resource usage by active users" expected to be low in this case?
{quote}
So, you know how I said that the above was a simplification? Here is another
wrinkle in that algorithm:
{noformat}
IF queueUsage <= queueCapacity
THEN
(total resources consumed by active users) =
yarn.scheduler.capacity.root.QueueA.capacity
ENDIF
So, basically, in the use case where QueueA is empty, the algorithm becomes:
{noformat}
max ((yarn.scheduler.capacity.root.QueueA.capacity) / (number of active users)),
((totalQueueResourcesAvailableToUsers) * min-user-limit-pct))
{noformat}
> User limit is not calculated as per definition for preemption
> -------------------------------------------------------------
>
> Key: YARN-10821
> URL: https://issues.apache.org/jira/browse/YARN-10821
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
> Reporter: Andras Gyori
> Assignee: Andras Gyori
> Priority: Major
> Attachments: YARN-10821.001.patch
>
>
> Minimum user limit percent (MULP) is a soft limit by definition. Preemption
> uses pending resources to determine the resources needed by a queue, which is
> calculated in LeafQueue#getTotalPendingResourcesConsideringUserLimit. This
> method involves headroom calculated by UsersManager#computeUserLimit.
> However, the pending resources for preemption are limited in an unexpected
> fashion.
> * In LeafQueue#getUserAMResourceLimitPerPartition an effective userLimit is
> calculated first:
> {code:java}
> float effectiveUserLimit = Math.max(usersManager.getUserLimit() / 100.0f,
> 1.0f / Math.max(getAbstractUsersManager().getNumActiveUsers(), 1));
> {code}
> * In UsersManager#computeUserLimit the userLimit is calculated as is
> (currentCapacity * userLimit)
> {code:java}
> Resource userLimitResource = Resources.max(resourceCalculator,
> partitionResource,
> Resources.divideAndCeil(resourceCalculator, resourceUsed,
> usersSummedByWeight),
> Resources.divideAndCeil(resourceCalculator,
> Resources.multiplyAndRoundDown(currentCapacity, getUserLimit()),
> 100));
> {code}
> The fewer users occupying the queue, the more prevalent and outstanding this
> effect will be in preemption.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
