[
https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14224034#comment-14224034
]
Craig Welch commented on YARN-2637:
-----------------------------------
One open question still in my mind is whether or not the configuration
parameter should be changed to actually behave as a "percent". Other things so
named (userlimit, at least) are actually a percentage - and the name of this
parameter tends to suggest that - but it is actually just a float value (so you
would use .1 to limit to 10 percent of cluster resource, not 10...). I did
take a pass at making the change, it looks doable (with quite a few more test
changes...). On the one hand, it seems like the time to make this change, as
the meaning of the value is changing considerably as it is. On the other hand,
it may be more impact than we want - as users who have configured, say, .3,
will still have about the same behavior on a sizable cluster as they do today
with the change as it is now, but if we modify it to actually behave as a
"percent" value (e.g. / 100), then it will have a far more limiting impact (if
the users do not adjust their configuration). Thoughts? Myself, I can see
arguments both ways, though I'm leaning toward making the change to remove all
of the "surprise" factor of how this parameter works... (e.g. make it a proper
% value)
> maximum-am-resource-percent could be violated when resource of AM is >
> minimumAllocation
> ----------------------------------------------------------------------------------------
>
> Key: YARN-2637
> URL: https://issues.apache.org/jira/browse/YARN-2637
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 2.6.0
> Reporter: Wangda Tan
> Assignee: Craig Welch
> Priority: Critical
> Attachments: YARN-2637.0.patch, YARN-2637.1.patch,
> YARN-2637.12.patch, YARN-2637.2.patch, YARN-2637.6.patch, YARN-2637.7.patch,
> YARN-2637.9.patch
>
>
> Currently, number of AM in leaf queue will be calculated in following way:
> {code}
> max_am_resource = queue_max_capacity * maximum_am_resource_percent
> #max_am_number = max_am_resource / minimum_allocation
> #max_am_number_for_each_user = #max_am_number * userlimit * userlimit_factor
> {code}
> And when submit new application to RM, it will check if an app can be
> activated in following way:
> {code}
> for (Iterator<FiCaSchedulerApp> i=pendingApplications.iterator();
> i.hasNext(); ) {
> FiCaSchedulerApp application = i.next();
>
> // Check queue limit
> if (getNumActiveApplications() >= getMaximumActiveApplications()) {
> break;
> }
>
> // Check user limit
> User user = getUser(application.getUser());
> if (user.getActiveApplications() <
> getMaximumActiveApplicationsPerUser()) {
> user.activateApplication();
> activeApplications.add(application);
> i.remove();
> LOG.info("Application " + application.getApplicationId() +
> " from user: " + application.getUser() +
> " activated in queue: " + getQueueName());
> }
> }
> {code}
> An example is,
> If a queue has capacity = 1G, max_am_resource_percent = 0.2, the maximum
> resource that AM can use is 200M, assuming minimum_allocation=1M, #am can be
> launched is 200, and if user uses 5M for each AM (> minimum_allocation). All
> apps can still be activated, and it will occupy all resource of a queue
> instead of only a max_am_resource_percent of a queue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
