[ https://issues.apache.org/jira/browse/YARN-4691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yufei Gu reassigned YARN-4691: ------------------------------ Assignee: Yufei Gu > Cache resource usage at FSLeafQueue level > ----------------------------------------- > > Key: YARN-4691 > URL: https://issues.apache.org/jira/browse/YARN-4691 > Project: Hadoop YARN > Issue Type: Improvement > Reporter: Ming Ma > Assignee: Yufei Gu > > As part of the fair share assignment, fair scheduler needs to sort queues to > decide which queue is furthest away from its fair share. During the sorting, > the comparator needs to get the Resource usage of each queue. > The parent queue will aggregate the resource usage from leaf queues. The leaf > queue will aggregate the resource usage from all apps in the queue. > {noformat} > FSLeafQueue.java > @Override > public Resource getResourceUsage() { > Resource usage = Resources.createResource(0); > readLock.lock(); > try { > for (FSAppAttempt app : runnableApps) { > Resources.addTo(usage, app.getResourceUsage()); > } > for (FSAppAttempt app : nonRunnableApps) { > Resources.addTo(usage, app.getResourceUsage()); > } > } finally { > readLock.unlock(); > } > return usage; > } > {noformat} > Each time fair scheduler tries to assign a container, it needs to sort all > queues. Thus the number of Resources.addTo operations will be > (number_of_queues) * lg(number_of_queues) * number_of_apps_per_queue, or > number_of_apps_on_the_cluster * lg(number_of_queues). > One way to solve this is to cache the resource usage at FSLeafQueue level. > Each time fair scheduler updates FSAppAttempt's resource usage, it will > update FSLeafQueue resource usage. This will greatly reduce the overall > number of Resources.addTo operations. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org