Craig Welch commented on YARN-2637:

Regarding the findbugs report for LeafQueue.lastClusterResource - access to 
lastClusterResource appears to be synchronized everywhere except 
getAbsActualCapacity, which I don't actually see being used anywhere - I'm 
going to add a findbugs exception and a comment on the method so that if it is 
used in the future synchronization can be addressed

-re [~leftnoteasy] 's latest:

-re 1 - actually, user limits are based on absolute queue capacity rather than 
max capacity - this is apparently intentional because, although a queue can 
exceed it's absolute capacity, an individual user is not supposed to, hence my 
basing the user amlimit on the absolute capacity.  The approach I use fits with 
the original logic in CSQueueUtils which allows a user the greater of the 
userlimit share of the absolute capacity or 1/# active users (so if there are 
fewer users active than would reach the userlimit they can use the full queue 
absolute capacity), the only correction being that we are using the actual 
value of resources by application masters instead of one based on minalloc

-re 2 - Actually, the snippet provided is not quite correct, some schedulers 
provide a cpu value as well.  In any case, for encapsulation reasons it's 
better to use the scheduler's value in case its means of determining this 
changes in the future. 

-re 3 - I can't see this making the slightest difference in understandability - 
since these test's paths don't populate the rmapps I would simply be 
individually putting mocked ones into the map instead of the single mock + 
matcher for all the apps.  The way it is seems clearer to me as all of the 
mocking is together instead of distributing the (mock activity, if not mock 
framework...) process of putting mock rmapps into the collection throughout the 

-re 4 - interesting, those were already there, but I also couldn't see why.  
Test passes fine without them, so I removed them

-re 5 - removed

uploading updated patch in a few

> maximum-am-resource-percent could be respected for both LeafQueue/User when 
> trying to activate applications.
> ------------------------------------------------------------------------------------------------------------
>                 Key: YARN-2637
>                 URL: https://issues.apache.org/jira/browse/YARN-2637
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager
>    Affects Versions: 2.6.0
>            Reporter: Wangda Tan
>            Assignee: Craig Welch
>            Priority: Critical
>         Attachments: YARN-2637.0.patch, YARN-2637.1.patch, 
> YARN-2637.12.patch, YARN-2637.13.patch, YARN-2637.15.patch, 
> YARN-2637.16.patch, YARN-2637.17.patch, YARN-2637.18.patch, 
> YARN-2637.19.patch, YARN-2637.2.patch, YARN-2637.20.patch, 
> YARN-2637.21.patch, YARN-2637.22.patch, YARN-2637.23.patch, 
> YARN-2637.25.patch, YARN-2637.26.patch, YARN-2637.27.patch, 
> YARN-2637.28.patch, YARN-2637.29.patch, YARN-2637.30.patch, 
> YARN-2637.31.patch, YARN-2637.32.patch, YARN-2637.36.patch, 
> YARN-2637.38.patch, YARN-2637.39.patch, YARN-2637.6.patch, YARN-2637.7.patch, 
> YARN-2637.9.patch
> Currently, number of AM in leaf queue will be calculated in following way:
> {code}
> max_am_resource = queue_max_capacity * maximum_am_resource_percent
> #max_am_number = max_am_resource / minimum_allocation
> #max_am_number_for_each_user = #max_am_number * userlimit * userlimit_factor
> {code}
> And when submit new application to RM, it will check if an app can be 
> activated in following way:
> {code}
>     for (Iterator<FiCaSchedulerApp> i=pendingApplications.iterator(); 
>          i.hasNext(); ) {
>       FiCaSchedulerApp application = i.next();
>       // Check queue limit
>       if (getNumActiveApplications() >= getMaximumActiveApplications()) {
>         break;
>       }
>       // Check user limit
>       User user = getUser(application.getUser());
>       if (user.getActiveApplications() < 
> getMaximumActiveApplicationsPerUser()) {
>         user.activateApplication();
>         activeApplications.add(application);
>         i.remove();
>         LOG.info("Application " + application.getApplicationId() +
>             " from user: " + application.getUser() + 
>             " activated in queue: " + getQueueName());
>       }
>     }
> {code}
> An example is,
> If a queue has capacity = 1G, max_am_resource_percent  = 0.2, the maximum 
> resource that AM can use is 200M, assuming minimum_allocation=1M, #am can be 
> launched is 200, and if user uses 5M for each AM (> minimum_allocation). All 
> apps can still be activated, and it will occupy all resource of a queue 
> instead of only a max_am_resource_percent of a queue.

This message was sent by Atlassian JIRA

Reply via email to