Kannan Rajah commented on YARN-2989:

I am going to close this bug since the load balancing is already implemented in 
YARN-1290. But I think we can improve the performance of logic in 
continuousScheduling when Label Based Scheduling is active. 

for each node (ordered by cap remaining)
  for each schedulable (ordered by fairness)
    if a set of conditions are met
      assign the container to node

When LBS is enabled, the set of conditions will include the label match. A node 
with maximum capacity remaining may not meet the label criteria. So why bother 
going through a global set of nodes when only a subset of them can even be used 
to schedule that application. What we really need is to track a set of sub 
clusters and the applications that can be scheduled on them. Within each sub 
cluster, we will maintain this node ordering by capacity remaining. Note that a 
cluster is also a subcluster.

for each subcluster
  if there are no applications belonging to it

  for each node in the subcluster (ordered by cap remaining)
    for each schedulable (ordered by fairness)
      if a set of conditions are met
        assign the container to node

I will track this problem in YARN-796.

> Better Load Balancing in Fair Scheduler
> ---------------------------------------
>                 Key: YARN-2989
>                 URL: https://issues.apache.org/jira/browse/YARN-2989
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: fairscheduler
>    Affects Versions: 2.5.0
>            Reporter: Kannan Rajah
> While porting Fair Scheduler from MR1, we seem to have changed the logic 
> behind task distribution across nodes (MAPREDUCE-3451).
> In MR1, a load factor was computed using runnableMaps/totalMapSlots and this 
> was used to determine how many tasks need to be given to a node such that the 
> overall cluster load is evenly distributed. In one heartbeat, we could assign 
> multiple tasks. In YARN, we have the option to assign multiple tasks to a 
> node, but this is disabled by default (YARN-302). Even when it is enabled, 
> the number of tasks to assign is statically configured. So it won't ensure 
> that load is evenly distributed. Why not bring back the load factor based 
> check? Any reason why it was not done? This is actually more relevant with 
> label based scheduling.
> If there are no objections, I would like to implement it for both normal and 
> label based scheduling scenarios.

This message was sent by Atlassian JIRA

Reply via email to