[
https://issues.apache.org/jira/browse/YARN-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated YARN-5892:
-----------------------------
Attachment: YARN-5892.009.patch
Thanks [~sunilg] for your comments.
bq. 1. {{getNumActive/AllWeightedUsers}} is iterating on all users now. It
could be optimized by considering only weighted users.
The method is probably named badly. This method really does need to iterate
over all users to determine the sum of all the weights. There may be
optimizations possible, but since these methods are only called from
{{computeUserLimit}}, they are only recomputed when necessary. I renamed the
methods to {{countActiveUsersTimesWeights}} and {{countAllUsersTimesWeights}}.
{quote}
2.Given we have only one user in cluster, and we gave UL < 1 for that user.
This will be increase the defined UserLimit for that user, because
getNumActive/AllWeightedUsers will give a count < 1 in this case. Correct?
{quote}
The return value of {{computeUserLimit}} is {{userLimitResource}}. If the sum
of active user(s)'s weight(s) is < 1, then it is true that
{{userLimitResource}} is larger than the actual number of resources used,
sometimes much larger. However, this works out fine because the only methods
that ever use the value of {{userLimitResource}} are
{{getComputedResourceLimitFor\[Active|All\]Users}}. These methods then multiply
the value of {{userLimitResource}} by the appropriate user weight(s), which
will result in the correct value of {{userLimit}} for that specific user.
bq. 3.I think user_weight=0 is tough to go in with existing nomenclatures or
usages. If AM container is huge, still we may give resource for that user.
Yes, the _user weight of 0_ case is a special one that we would like to
explore, but certainly not in this JIRA. Weight of 0 would have a special
meaning that although the user will only be given resources if no one else is
asking, that user can still use the whole queue.
> Capacity Scheduler: Support user-specific minimum user limit percent
> --------------------------------------------------------------------
>
> Key: YARN-5892
> URL: https://issues.apache.org/jira/browse/YARN-5892
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
> Reporter: Eric Payne
> Assignee: Eric Payne
> Attachments: Active users highlighted.jpg, YARN-5892.001.patch,
> YARN-5892.002.patch, YARN-5892.003.patch, YARN-5892.004.patch,
> YARN-5892.005.patch, YARN-5892.006.patch, YARN-5892.007.patch,
> YARN-5892.008.patch, YARN-5892.009.patch
>
>
> Currently, in the capacity scheduler, the {{minimum-user-limit-percent}}
> property is per queue. A cluster admin should be able to set the minimum user
> limit percent on a per-user basis within the queue.
> This functionality is needed so that when intra-queue preemption is enabled
> (YARN-4945 / YARN-2113), some users can be deemed as more important than
> other users, and resources from VIP users won't be as likely to be preempted.
> For example, if the {{getstuffdone}} queue has a MULP of 25 percent, but user
> {{jane}} is a power user of queue {{getstuffdone}} and needs to be guaranteed
> 75 percent, the properties for {{getstuffdone}} and {{jane}} would look like
> this:
> {code}
> <property>
>
> <name>yarn.scheduler.capacity.root.getstuffdone.minimum-user-limit-percent</name>
> <value>25</value>
> </property>
> <property>
>
> <name>yarn.scheduler.capacity.root.getstuffdone.jane.minimum-user-limit-percent</name>
> <value>75</value>
> </property>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
