Andras Gyori created YARN-10821: ----------------------------------- Summary: User limit is not calculated as per definition for preemption Key: YARN-10821 URL: https://issues.apache.org/jira/browse/YARN-10821 Project: Hadoop YARN Issue Type: Bug Components: capacity scheduler Reporter: Andras Gyori Assignee: Andras Gyori
Minimum user limit percent (MULP) is a soft limit by definition. Preemption uses pending resources to determine the resources needed by a queue, which is calculated in LeafQueue#getTotalPendingResourcesConsideringUserLimit. This method involves headroom calculated by UsersManager#computeUserLimit. However, the pending resources for preemption are limited in an unexpected fashion. * In LeafQueue#getUserAMResourceLimitPerPartition an effective userLimit is calculated first: {code:java} float effectiveUserLimit = Math.max(usersManager.getUserLimit() / 100.0f, 1.0f / Math.max(getAbstractUsersManager().getNumActiveUsers(), 1)); {code} * In UsersManager#computeUserLimit the userLimit is calculated as is (currentCapacity * userLimit) {code:java} Resource userLimitResource = Resources.max(resourceCalculator, partitionResource, Resources.divideAndCeil(resourceCalculator, resourceUsed, usersSummedByWeight), Resources.divideAndCeil(resourceCalculator, Resources.multiplyAndRoundDown(currentCapacity, getUserLimit()), 100)); {code} The fewer users occupying the queue, the more prevalent and outstanding this effect will be in preemption. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org