[
https://issues.apache.org/jira/browse/YARN-4691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu reassigned YARN-4691:
------------------------------
Assignee: Yufei Gu
> Cache resource usage at FSLeafQueue level
> -----------------------------------------
>
> Key: YARN-4691
> URL: https://issues.apache.org/jira/browse/YARN-4691
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: Ming Ma
> Assignee: Yufei Gu
>
> As part of the fair share assignment, fair scheduler needs to sort queues to
> decide which queue is furthest away from its fair share. During the sorting,
> the comparator needs to get the Resource usage of each queue.
> The parent queue will aggregate the resource usage from leaf queues. The leaf
> queue will aggregate the resource usage from all apps in the queue.
> {noformat}
> FSLeafQueue.java
> @Override
> public Resource getResourceUsage() {
> Resource usage = Resources.createResource(0);
> readLock.lock();
> try {
> for (FSAppAttempt app : runnableApps) {
> Resources.addTo(usage, app.getResourceUsage());
> }
> for (FSAppAttempt app : nonRunnableApps) {
> Resources.addTo(usage, app.getResourceUsage());
> }
> } finally {
> readLock.unlock();
> }
> return usage;
> }
> {noformat}
> Each time fair scheduler tries to assign a container, it needs to sort all
> queues. Thus the number of Resources.addTo operations will be
> (number_of_queues) * lg(number_of_queues) * number_of_apps_per_queue, or
> number_of_apps_on_the_cluster * lg(number_of_queues).
> One way to solve this is to cache the resource usage at FSLeafQueue level.
> Each time fair scheduler updates FSAppAttempt's resource usage, it will
> update FSLeafQueue resource usage. This will greatly reduce the overall
> number of Resources.addTo operations.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
