[ 
https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14268060#comment-14268060
 ] 

Wangda Tan commented on YARN-2637:
----------------------------------

Hi [~cwelch],
Thanks for updating, it's glad to see not too much code added to support 
max-am-resource-percent for user.

Some comments:

1. In LeafQueue, 

1.1 This check is not needed, right? Since we already counted 
consumedAMResource in each user:
{code}
      if (user.getActiveApplications() < getMaximumActiveApplicationsPerUser()) 
{
{code}

1.2 {{maxActiveAppsUsingAbsCap}} and {{maxActiveApplicationsPerUser}} could be 
removed?

1.3 Such log when activating in queue/user will be very frequent, using {{if 
debug-enabled, LOG.debug}} instead? 
{code}
+          LOG.info("not starting application as amIfStarted exceeds amLimit");
+          continue;
{code}
And print amIfStart/amLimit resource with the log instead of just says it 
exceeds?

1.4 Comment should be "// Check user AM resource limit?" 
{code}
+      // Check user limits
       User user = getUser(application.getUser());
+      
+      // AM Resource Limit
{code}

2. Test
Similar to what I suggested for test queue AM resource limit, 
{{testUserAMResourceLimitAccumulated}} should be really "accumulated", test 
case should contain >1 app allowed for each user.

Wangda

> maximum-am-resource-percent could be violated when resource of AM is > 
> minimumAllocation
> ----------------------------------------------------------------------------------------
>
>                 Key: YARN-2637
>                 URL: https://issues.apache.org/jira/browse/YARN-2637
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager
>    Affects Versions: 2.6.0
>            Reporter: Wangda Tan
>            Assignee: Craig Welch
>            Priority: Critical
>         Attachments: YARN-2637.0.patch, YARN-2637.1.patch, 
> YARN-2637.12.patch, YARN-2637.13.patch, YARN-2637.15.patch, 
> YARN-2637.16.patch, YARN-2637.17.patch, YARN-2637.18.patch, 
> YARN-2637.19.patch, YARN-2637.2.patch, YARN-2637.20.patch, 
> YARN-2637.21.patch, YARN-2637.22.patch, YARN-2637.23.patch, 
> YARN-2637.25.patch, YARN-2637.26.patch, YARN-2637.27.patch, 
> YARN-2637.28.patch, YARN-2637.29.patch, YARN-2637.30.patch, 
> YARN-2637.6.patch, YARN-2637.7.patch, YARN-2637.9.patch
>
>
> Currently, number of AM in leaf queue will be calculated in following way:
> {code}
> max_am_resource = queue_max_capacity * maximum_am_resource_percent
> #max_am_number = max_am_resource / minimum_allocation
> #max_am_number_for_each_user = #max_am_number * userlimit * userlimit_factor
> {code}
> And when submit new application to RM, it will check if an app can be 
> activated in following way:
> {code}
>     for (Iterator<FiCaSchedulerApp> i=pendingApplications.iterator(); 
>          i.hasNext(); ) {
>       FiCaSchedulerApp application = i.next();
>       
>       // Check queue limit
>       if (getNumActiveApplications() >= getMaximumActiveApplications()) {
>         break;
>       }
>       
>       // Check user limit
>       User user = getUser(application.getUser());
>       if (user.getActiveApplications() < 
> getMaximumActiveApplicationsPerUser()) {
>         user.activateApplication();
>         activeApplications.add(application);
>         i.remove();
>         LOG.info("Application " + application.getApplicationId() +
>             " from user: " + application.getUser() + 
>             " activated in queue: " + getQueueName());
>       }
>     }
> {code}
> An example is,
> If a queue has capacity = 1G, max_am_resource_percent  = 0.2, the maximum 
> resource that AM can use is 200M, assuming minimum_allocation=1M, #am can be 
> launched is 200, and if user uses 5M for each AM (> minimum_allocation). All 
> apps can still be activated, and it will occupy all resource of a queue 
> instead of only a max_am_resource_percent of a queue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to