[
https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14236389#comment-14236389
]
Craig Welch commented on YARN-2637:
-----------------------------------
Something to be aware of wrt the way I implemented option 2- when making the
change I noticed that, in fact, the share of the maxampercent apportioned out
to queues was actually based on the maximum capacity of the queue
(absMaxCapacity) not the baseline value (absCapacity). I went ahead and kept
this logic because it is consistent with the earlier approach and it still
provides the desired control although potentially allowing more resources being
provided to am's in cluster-aggregate if queues have high max values. Meaning
if the total max is say 200% (sum of queue max, as opposed to the 100%
baseline), then the standard .1 maxam will actually allow 20% usage of
resources for application masters in the cluster. You can still manage the
usage effectively, though, so I thought it best to stay as consistent with
existing meaning as possible (while fixing the surprising aspect of it...),
just wanted to make sure you were aware/it was documented
> maximum-am-resource-percent could be violated when resource of AM is >
> minimumAllocation
> ----------------------------------------------------------------------------------------
>
> Key: YARN-2637
> URL: https://issues.apache.org/jira/browse/YARN-2637
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 2.6.0
> Reporter: Wangda Tan
> Assignee: Craig Welch
> Priority: Critical
> Attachments: YARN-2637.0.patch, YARN-2637.1.patch,
> YARN-2637.12.patch, YARN-2637.13.patch, YARN-2637.15.patch,
> YARN-2637.16.patch, YARN-2637.2.patch, YARN-2637.6.patch, YARN-2637.7.patch,
> YARN-2637.9.patch
>
>
> Currently, number of AM in leaf queue will be calculated in following way:
> {code}
> max_am_resource = queue_max_capacity * maximum_am_resource_percent
> #max_am_number = max_am_resource / minimum_allocation
> #max_am_number_for_each_user = #max_am_number * userlimit * userlimit_factor
> {code}
> And when submit new application to RM, it will check if an app can be
> activated in following way:
> {code}
> for (Iterator<FiCaSchedulerApp> i=pendingApplications.iterator();
> i.hasNext(); ) {
> FiCaSchedulerApp application = i.next();
>
> // Check queue limit
> if (getNumActiveApplications() >= getMaximumActiveApplications()) {
> break;
> }
>
> // Check user limit
> User user = getUser(application.getUser());
> if (user.getActiveApplications() <
> getMaximumActiveApplicationsPerUser()) {
> user.activateApplication();
> activeApplications.add(application);
> i.remove();
> LOG.info("Application " + application.getApplicationId() +
> " from user: " + application.getUser() +
> " activated in queue: " + getQueueName());
> }
> }
> {code}
> An example is,
> If a queue has capacity = 1G, max_am_resource_percent = 0.2, the maximum
> resource that AM can use is 200M, assuming minimum_allocation=1M, #am can be
> launched is 200, and if user uses 5M for each AM (> minimum_allocation). All
> apps can still be activated, and it will occupy all resource of a queue
> instead of only a max_am_resource_percent of a queue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
