[
https://issues.apache.org/jira/browse/YARN-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated YARN-5892:
-----------------------------
Fix Version/s: (was: 3.0.0-beta1)
3.0.0-alpha3
[~andrew.wang], this JIRA is already in 3.0.0.alpha4 (revision
ca13b224b2feb9c44de861da9cbba8dd2a12cb35). I reopened this JIRA so that I could
implement the backport to branches 2 and 2.8.
[~sunilg], yes, if we could get YARN-3140 backported to branch-2, that would
indeed be the best. However, I am worried about the time it will take to do
that. I don't think I have the bandwidth to do that right now. Are you
volunteering to do the backport to branch-2?
Regarding locking for {{activeUsersSet}}, I don't think it is necessary if we
can overcome the concurrent modification exception problem. In fact, simple
synchronization on this method will cause deadlocks, and I think read/write
locks will have the same problem, if on a smaller scale.
I don't think locking is necessary. If the sum is inaccurate, it will be
recomputed during the next round because the list of active users has changed.
On a related note (if no locking is implemented), I think the following logic
is incorrect:
{code}
for (String userName : activeUsersSet) {
// Do the following instead of calling getUser so locking is not needed.
User user = users.get(userName);
count += (user != null) ? user.getWeight() : 1.0f;
}
{code}
I think that if the user was in {{activeUsersSet}} when the for loop started
but was later removed from the {{users}} map, the user weight should be 0.0f
instead of 1.0f
> Support user-specific minimum user limit percentage in Capacity Scheduler
> -------------------------------------------------------------------------
>
> Key: YARN-5892
> URL: https://issues.apache.org/jira/browse/YARN-5892
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
> Reporter: Eric Payne
> Assignee: Eric Payne
> Fix For: 3.0.0-alpha3
>
> Attachments: Active users highlighted.jpg, YARN-5892.001.patch,
> YARN-5892.002.patch, YARN-5892.003.patch, YARN-5892.004.patch,
> YARN-5892.005.patch, YARN-5892.006.patch, YARN-5892.007.patch,
> YARN-5892.008.patch, YARN-5892.009.patch, YARN-5892.010.patch,
> YARN-5892.012.patch, YARN-5892.013.patch, YARN-5892.014.patch,
> YARN-5892.015.patch, YARN-5892.branch-2.015.patch
>
>
> Currently, in the capacity scheduler, the {{minimum-user-limit-percent}}
> property is per queue. A cluster admin should be able to set the minimum user
> limit percent on a per-user basis within the queue.
> This functionality is needed so that when intra-queue preemption is enabled
> (YARN-4945 / YARN-2113), some users can be deemed as more important than
> other users, and resources from VIP users won't be as likely to be preempted.
> For example, if the {{getstuffdone}} queue has a MULP of 25 percent, but user
> {{jane}} is a power user of queue {{getstuffdone}} and needs to be guaranteed
> 75 percent, the properties for {{getstuffdone}} and {{jane}} would look like
> this:
> {code}
> <property>
>
> <name>yarn.scheduler.capacity.root.getstuffdone.minimum-user-limit-percent</name>
> <value>25</value>
> </property>
> <property>
>
> <name>yarn.scheduler.capacity.root.getstuffdone.jane.minimum-user-limit-percent</name>
> <value>75</value>
> </property>
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
