[ 
https://issues.apache.org/jira/browse/YARN-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14535215#comment-14535215
 ] 

Chen He commented on YARN-1612:
-------------------------------

Hi [~kasha], thank you for the quick reply. I can explain this:

1. this JIRA is to enable delay scheduling in default. It means if user does 
not specify we should enable delay scheduling. If user set the delay to be "0" 
in xml file, we should allow user to disable delay scheduling. 

2. The Fairscheduler  enable delay through following code:
    FSSchedulerApp.java
    // if not being used, can schedule anywhere
    if (nodeLocalityDelayMs < 0 || rackLocalityDelayMs < 0) {
      return NodeType.OFF_SWITCH;
    }
 Since default nodeLocalityDelayMs and rackLocalityDelayMs are both "-1L', to 
enable delay algorithm in default, we need to change 
"DEFAULT_LOCALITY_DELAY_NODE_MS" and "DEFAULT_LOCALITY_DELAY_RACK_MS" to be a 
number what larger than "0". 

3. What default delay value should we choose? This is important :
{quote}
Since the delay interval will benefits the data locality but also affects map 
tasks assignment if it is too long, to enable delay scheduling by default, we 
need a relative reasonable delay interval;
{quote}

In [~matei]'s delay algorithm paper, he suggested the delay interval should be 
3 times of heartbeat interval to achieve best performance (based on Facebook 
workload). Then, I make it to be "3* 
YarnConfiguration.DEFAULT_RM_NM_HEARTBEAT_INTERVAL_MS"

> FairScheduler: Enable delay scheduling by default
> -------------------------------------------------
>
>                 Key: YARN-1612
>                 URL: https://issues.apache.org/jira/browse/YARN-1612
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: fairscheduler
>            Reporter: Sandy Ryza
>            Assignee: Chen He
>         Attachments: YARN-1612-003.patch, YARN-1612-004.patch, 
> YARN-1612-v2.patch, YARN-1612.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to