[
https://issues.apache.org/jira/browse/YARN-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16032452#comment-16032452
]
daemon commented on YARN-6320:
------------------------------
I think I have solved the problem and in my test environment the scheduler can
assign over 5000 containers per second.
> FairScheduler:Identifying apps to assign in updateThread
> --------------------------------------------------------
>
> Key: YARN-6320
> URL: https://issues.apache.org/jira/browse/YARN-6320
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Tao Jie
>
> In FairScheduler today, we have 1)UpdateThread that update queue/app status,
> fairshare, starvation info, 2)nodeUpdate triggered by NM heartbeat that do
> the scheduling. When we handle one nodeUpdate, we will top-down from the root
> queue to the leafqueues and find the most needy application to allocate
> container according to queue's fairshare. Also we should sort children at
> each hierarchy level.
> My thought is that we have a global sorted {{candidateAppList}} which keeps
> apps need to assign, and move the logic that "find app that should allocate
> resource to" from nodeUpdate to UpdateThread. In UpdateThread, we find
> candidate apps to assign and put them into {{candidateAppList}}. In
> nodeUpdate, we consume the list and allocate containers to apps.
> As far as I see, we can have 3 benifits:
> 1, nodeUpdate() is invoked much more frequently than update() in
> UpdateThread, especially in a large cluster. As a result we can reduce much
> unnecessary sorting.
> 2, It will have better coordination with YARN-5829, we can indicate apps to
> assign more directly rather than let nodes find the best apps to assign.
> 3, It seems to be easier to introduce scheduling restricts such as nodelabel,
> affinity/anti-affinity into FS, since we can pre-allocate containers
> asynchronously.
> [~kasha], [~templedf], [~yufeigu] like to hear your thoughts.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
