[ https://issues.apache.org/jira/browse/YARN-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14274029#comment-14274029 ]
Wangda Tan commented on YARN-2637: ---------------------------------- [~cwelch], Thanks for updating, reviewed latest patch, some comments: *LeafQueue.java:* 1) getUserAMResourceLimit() I think according to how we compute user-limit in LeafQueue, should we do it following way to compute user-am-resource-limit? {code} user-am-resource-limit = am-resource-percent * min( queue-max-capacity * max(user-limit, 1/#active-user), queue-configured-capacity * user-limit-factor)? {code} Thoughts? *FiCaSchedulerApp.java:* 2) Is it necessary to get scheduler instance just for minimum allocation, do you think is it better to just get minimum allocation using: {code} minAllocMb = rmContext.getConf().getInt( YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB, YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB); {code} Which can avoid creating mocked scheduler in some of test changes, including changes in RMContext.java. *TestApplicationLimits:* 3) I think RMContext.getRMApps can be updated directly, it's no need to spy it, I suggest avoid spying objects in tests as much as we can. It's not very understandable and easily cause problems when we update implementations. 4) Such spy invokes are unnecessary? {code} FiCaSchedulerApp app_0_0 = spy(new FiCaSchedulerApp(appAttemptId_0_0, user_0, queue, {code} 5) Nobody is using it, should remove it. {code} private FiCaSchedulerApp getMockApplication(int appId, String user) { return getMockApplication(appId, user, Resource.newInstance(0, 0)); } {code} Wangda > maximum-am-resource-percent could be respected for both LeafQueue/User when > trying to activate applications. > ------------------------------------------------------------------------------------------------------------ > > Key: YARN-2637 > URL: https://issues.apache.org/jira/browse/YARN-2637 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager > Affects Versions: 2.6.0 > Reporter: Wangda Tan > Assignee: Craig Welch > Priority: Critical > Attachments: YARN-2637.0.patch, YARN-2637.1.patch, > YARN-2637.12.patch, YARN-2637.13.patch, YARN-2637.15.patch, > YARN-2637.16.patch, YARN-2637.17.patch, YARN-2637.18.patch, > YARN-2637.19.patch, YARN-2637.2.patch, YARN-2637.20.patch, > YARN-2637.21.patch, YARN-2637.22.patch, YARN-2637.23.patch, > YARN-2637.25.patch, YARN-2637.26.patch, YARN-2637.27.patch, > YARN-2637.28.patch, YARN-2637.29.patch, YARN-2637.30.patch, > YARN-2637.31.patch, YARN-2637.32.patch, YARN-2637.36.patch, > YARN-2637.38.patch, YARN-2637.39.patch, YARN-2637.6.patch, YARN-2637.7.patch, > YARN-2637.9.patch > > > Currently, number of AM in leaf queue will be calculated in following way: > {code} > max_am_resource = queue_max_capacity * maximum_am_resource_percent > #max_am_number = max_am_resource / minimum_allocation > #max_am_number_for_each_user = #max_am_number * userlimit * userlimit_factor > {code} > And when submit new application to RM, it will check if an app can be > activated in following way: > {code} > for (Iterator<FiCaSchedulerApp> i=pendingApplications.iterator(); > i.hasNext(); ) { > FiCaSchedulerApp application = i.next(); > > // Check queue limit > if (getNumActiveApplications() >= getMaximumActiveApplications()) { > break; > } > > // Check user limit > User user = getUser(application.getUser()); > if (user.getActiveApplications() < > getMaximumActiveApplicationsPerUser()) { > user.activateApplication(); > activeApplications.add(application); > i.remove(); > LOG.info("Application " + application.getApplicationId() + > " from user: " + application.getUser() + > " activated in queue: " + getQueueName()); > } > } > {code} > An example is, > If a queue has capacity = 1G, max_am_resource_percent = 0.2, the maximum > resource that AM can use is 200M, assuming minimum_allocation=1M, #am can be > launched is 200, and if user uses 5M for each AM (> minimum_allocation). All > apps can still be activated, and it will occupy all resource of a queue > instead of only a max_am_resource_percent of a queue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)