[
https://issues.apache.org/jira/browse/YARN-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951539#comment-15951539
]
Wangda Tan commented on YARN-5892:
----------------------------------
Thanks [~eepayne] to update the patch.
Several comments:
1) For User weight:
- I prefer to only allow specify it in the leaf queue level. Inheriting from
parent causes confusing in other cases. (For example should we allow
overwriting in leaf, etc.)
- I think we should limit the user weight to be: {{User_weight <= 100 / MULP}}
when validating the config. Otherwise if MULP = 100, and user X's weight = 3,
by definition it can get guaranteed resource of 3 * queue configured resource,
which is not good.
2) LeafQueue:
- getUserAMResourceLimitPerPartition(String) is not used by anyone, can be
removed.
3) UsersManager:
I think logics in the {{computeUserLimit}} should be updated:
{code}
int usersCount = getNumActiveUsers();
Resource resourceUsed = totalResUsageForActiveUsers.getUsed(nodePartition);
// For non-activeUser calculation, consider all users count.
if (!activeUser) {
resourceUsed = currentCapacity;
usersCount = users.size();
}
{code}
Users_count should be a float and impacted by weight according to definition of
weight (weight=x means equivalent to x users). For example, if a two users in
the queue, one has weight=2, one has weight = 0.5, the users count should be
2.5. And we need get separately users_count_by_weight for all users and active
users.
4) updateUsersWeights:
Should we move userLimitNeedsRecompute into the weight lock, otherwise it could
cause a race condition.
5) Could you update doc with more detailed example?
> Capacity Scheduler: Support user-specific minimum user limit percent
> --------------------------------------------------------------------
>
> Key: YARN-5892
> URL: https://issues.apache.org/jira/browse/YARN-5892
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
> Reporter: Eric Payne
> Assignee: Eric Payne
> Attachments: Active users highlighted.jpg, YARN-5892.001.patch,
> YARN-5892.002.patch, YARN-5892.003.patch, YARN-5892.004.patch,
> YARN-5892.005.patch, YARN-5892.006.patch
>
>
> Currently, in the capacity scheduler, the {{minimum-user-limit-percent}}
> property is per queue. A cluster admin should be able to set the minimum user
> limit percent on a per-user basis within the queue.
> This functionality is needed so that when intra-queue preemption is enabled
> (YARN-4945 / YARN-2113), some users can be deemed as more important than
> other users, and resources from VIP users won't be as likely to be preempted.
> For example, if the {{getstuffdone}} queue has a MULP of 25 percent, but user
> {{jane}} is a power user of queue {{getstuffdone}} and needs to be guaranteed
> 75 percent, the properties for {{getstuffdone}} and {{jane}} would look like
> this:
> {code}
> <property>
>
> <name>yarn.scheduler.capacity.root.getstuffdone.minimum-user-limit-percent</name>
> <value>25</value>
> </property>
> <property>
>
> <name>yarn.scheduler.capacity.root.getstuffdone.jane.minimum-user-limit-percent</name>
> <value>75</value>
> </property>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
