[
https://issues.apache.org/jira/browse/YARN-6361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131803#comment-16131803
]
YunFan Zhou commented on YARN-6361:
-----------------------------------
[~yufeigu] Thank you Yufei.
It was in our production environment before, and I made a tradeoff in ensuring
equitable sharing and scheduling performance, sacrificing part of equitable
sharing but providing scheduling performance.
As you say, scheduling is a very generic component, so we should ensure
fairness as possible as we can.
So I later will try to come up with other sacrifice fairness and scheduling
performance point.
And it is obvious to see that every time after scheduling, in fact, the order
of the scheduling entity actually does not change too much, which can be based
on the optimization of the more completely.
In addition, the patch I submitted last time is just optimizing the performance
of the *fetchAppsWithDemand*, please help me review the code? Thank you.
> FairScheduler: FSLeafQueue.fetchAppsWithDemand CPU usage is high with big
> queues
> --------------------------------------------------------------------------------
>
> Key: YARN-6361
> URL: https://issues.apache.org/jira/browse/YARN-6361
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Miklos Szegedi
> Assignee: YunFan Zhou
> Attachments: dispatcherthread.png, threads.png, YARN-6361.001.patch
>
>
> FSLeafQueue.fetchAppsWithDemand sorts the applications by the current policy.
> Most of the time is spent in FairShareComparator.compare. We could improve
> this by doing the calculations outside the sort loop {{(O\(n\))}} and we
> sorted by a fixed number inside instead {{O(n*log\(n\))}}. This could be an
> performance issue when there are huge number of applications in a single
> queue. The attachments shows the performance impact when there are 10k
> applications in one queue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
