[
https://issues.apache.org/jira/browse/YARN-2989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14257841#comment-14257841
]
Kannan Rajah commented on YARN-2989:
------------------------------------
I am going to close this bug since the load balancing is already implemented in
YARN-1290. But I think we can improve the performance of logic in
continuousScheduling when Label Based Scheduling is active.
for each node (ordered by cap remaining)
for each schedulable (ordered by fairness)
if a set of conditions are met
assign the container to node
Problem:
When LBS is enabled, the set of conditions will include the label match. A node
with maximum capacity remaining may not meet the label criteria. So why bother
going through a global set of nodes when only a subset of them can even be used
to schedule that application. What we really need is to track a set of sub
clusters and the applications that can be scheduled on them. Within each sub
cluster, we will maintain this node ordering by capacity remaining. Note that a
cluster is also a subcluster.
for each subcluster
if there are no applications belonging to it
continue
for each node in the subcluster (ordered by cap remaining)
for each schedulable (ordered by fairness)
if a set of conditions are met
assign the container to node
I will track this problem in YARN-796.
> Better Load Balancing in Fair Scheduler
> ---------------------------------------
>
> Key: YARN-2989
> URL: https://issues.apache.org/jira/browse/YARN-2989
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
> Affects Versions: 2.5.0
> Reporter: Kannan Rajah
>
> While porting Fair Scheduler from MR1, we seem to have changed the logic
> behind task distribution across nodes (MAPREDUCE-3451).
> In MR1, a load factor was computed using runnableMaps/totalMapSlots and this
> was used to determine how many tasks need to be given to a node such that the
> overall cluster load is evenly distributed. In one heartbeat, we could assign
> multiple tasks. In YARN, we have the option to assign multiple tasks to a
> node, but this is disabled by default (YARN-302). Even when it is enabled,
> the number of tasks to assign is statically configured. So it won't ensure
> that load is evenly distributed. Why not bring back the load factor based
> check? Any reason why it was not done? This is actually more relevant with
> label based scheduling.
> If there are no objections, I would like to implement it for both normal and
> label based scheduling scenarios.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
