Andras Gyori created YARN-10821:
-----------------------------------
Summary: User limit is not calculated as per definition for
preemption
Key: YARN-10821
URL: https://issues.apache.org/jira/browse/YARN-10821
Project: Hadoop YARN
Issue Type: Bug
Components: capacity scheduler
Reporter: Andras Gyori
Assignee: Andras Gyori
Minimum user limit percent (MULP) is a soft limit by definition. Preemption
uses pending resources to determine the resources needed by a queue, which is
calculated in LeafQueue#getTotalPendingResourcesConsideringUserLimit. This
method involves headroom calculated by UsersManager#computeUserLimit. However,
the pending resources for preemption are limited in an unexpected fashion.
* In LeafQueue#getUserAMResourceLimitPerPartition an effective userLimit is
calculated first:
{code:java}
float effectiveUserLimit = Math.max(usersManager.getUserLimit() / 100.0f,
1.0f / Math.max(getAbstractUsersManager().getNumActiveUsers(), 1));
{code}
* In UsersManager#computeUserLimit the userLimit is calculated as is
(currentCapacity * userLimit)
{code:java}
Resource userLimitResource = Resources.max(resourceCalculator,
partitionResource,
Resources.divideAndCeil(resourceCalculator, resourceUsed,
usersSummedByWeight),
Resources.divideAndCeil(resourceCalculator,
Resources.multiplyAndRoundDown(currentCapacity, getUserLimit()),
100));
{code}
The fewer users occupying the queue, the more prevalent and outstanding this
effect will be in preemption.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
