Tao Jie created YARN-6320:
-----------------------------
Summary: FairScheduler:Identifying apps to assign in updateThread
Key: YARN-6320
URL: https://issues.apache.org/jira/browse/YARN-6320
Project: Hadoop YARN
Issue Type: Bug
Reporter: Tao Jie
In FairScheduler today, we have 1)UpdateThread that update queue/app status,
fairshare, starvation info, 2)nodeUpdate triggered by NM heartbeat that do the
scheduling. When we handle one nodeUpdate, we will top-down from the root queue
to the leafqueues and find the most needy application to allocate container
according to queue's fairshare. Also we should sort children at each hierarchy
level.
My thought is that we have a global sorted {{candidateAppList}} which keeps
apps need to assign, and move the logic that "find app that should allocate
resource to" from nodeUpdate to UpdateThread. In UpdateThread, we find
candidate apps to assign and put them into {{candidateAppList}}. In nodeUpdate,
we consume the list and allocate containers to apps.
As far as I see, we can have 3 benifits:
1, nodeUpdate() is invoked much more frequently than update() in UpdateThread,
especially in a large cluster. As a result we can reduce much unnecessary
sorting.
2, It will have better coordination with YARN-5829, we can indicate apps to
assign more directly rather than let nodes find the best apps to assign.
3, It seems to be easier to introduce scheduling restricts such as nodelabel,
affinity/anti-affinity into FS, since we can pre-allocate containers
asynchronously.
[~kasha], [~templedf], [~yufeigu] like to hear your thoughts.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
