[jira] [Commented] (YARN-1612) FairScheduler: Enable delay scheduling by default
[ https://issues.apache.org/jira/browse/YARN-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14543914#comment-14543914 ] Chen He commented on YARN-1612: --- Thank you for the review, [~kasha], I did change FairSchedulerConfiguration#DEFAULT_LOCALITY_THRESHOLD=1, but looks like it breaks other unit test. I will figure out the reason. > FairScheduler: Enable delay scheduling by default > - > > Key: YARN-1612 > URL: https://issues.apache.org/jira/browse/YARN-1612 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Reporter: Sandy Ryza >Assignee: Chen He > Attachments: YARN-1612-003.patch, YARN-1612-004.patch, > YARN-1612-v2.patch, YARN-1612.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1612) FairScheduler: Enable delay scheduling by default
[
https://issues.apache.org/jira/browse/YARN-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14535291#comment-14535291
]
Karthik Kambatla commented on YARN-1612:
[~airbots] - sorry, my bad. Missed this looking at to many patches at once.
Will slow down.
The patch should work when continuous scheduling is enabled. For the case when
it is disabled, we should likely enable it by setting
{{FairSchedulerConfiguration#DEFAULT_LOCALITY_THRESHOLD}} to a value > 0. May
be, a value of 1 would be comparable to setting the other value to thrice the
heartbeat.
> FairScheduler: Enable delay scheduling by default
> -
>
> Key: YARN-1612
> URL: https://issues.apache.org/jira/browse/YARN-1612
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Reporter: Sandy Ryza
>Assignee: Chen He
> Attachments: YARN-1612-003.patch, YARN-1612-004.patch,
> YARN-1612-v2.patch, YARN-1612.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1612) FairScheduler: Enable delay scheduling by default
[
https://issues.apache.org/jira/browse/YARN-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14535215#comment-14535215
]
Chen He commented on YARN-1612:
---
Hi [~kasha], thank you for the quick reply. I can explain this:
1. this JIRA is to enable delay scheduling in default. It means if user does
not specify we should enable delay scheduling. If user set the delay to be "0"
in xml file, we should allow user to disable delay scheduling.
2. The Fairscheduler enable delay through following code:
FSSchedulerApp.java
// if not being used, can schedule anywhere
if (nodeLocalityDelayMs < 0 || rackLocalityDelayMs < 0) {
return NodeType.OFF_SWITCH;
}
Since default nodeLocalityDelayMs and rackLocalityDelayMs are both "-1L', to
enable delay algorithm in default, we need to change
"DEFAULT_LOCALITY_DELAY_NODE_MS" and "DEFAULT_LOCALITY_DELAY_RACK_MS" to be a
number what larger than "0".
3. What default delay value should we choose? This is important :
{quote}
Since the delay interval will benefits the data locality but also affects map
tasks assignment if it is too long, to enable delay scheduling by default, we
need a relative reasonable delay interval;
{quote}
In [~matei]'s delay algorithm paper, he suggested the delay interval should be
3 times of heartbeat interval to achieve best performance (based on Facebook
workload). Then, I make it to be "3*
YarnConfiguration.DEFAULT_RM_NM_HEARTBEAT_INTERVAL_MS"
> FairScheduler: Enable delay scheduling by default
> -
>
> Key: YARN-1612
> URL: https://issues.apache.org/jira/browse/YARN-1612
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Reporter: Sandy Ryza
>Assignee: Chen He
> Attachments: YARN-1612-003.patch, YARN-1612-004.patch,
> YARN-1612-v2.patch, YARN-1612.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1612) FairScheduler: Enable delay scheduling by default
[ https://issues.apache.org/jira/browse/YARN-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14535027#comment-14535027 ] Karthik Kambatla commented on YARN-1612: In the latest patch, I only see changes that define two new DEFAULT values in FairScheduler. They aren't used anywhere. May be it is missing some changes? > FairScheduler: Enable delay scheduling by default > - > > Key: YARN-1612 > URL: https://issues.apache.org/jira/browse/YARN-1612 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Reporter: Sandy Ryza >Assignee: Chen He > Attachments: YARN-1612-003.patch, YARN-1612-004.patch, > YARN-1612-v2.patch, YARN-1612.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1612) FairScheduler: Enable delay scheduling by default
[
https://issues.apache.org/jira/browse/YARN-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14527820#comment-14527820
]
Hadoop QA commented on YARN-1612:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 33s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 31s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 35s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 50s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 1s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 14s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | yarn tests | 52m 36s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 88m 52s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12730302/YARN-1612-004.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 551615f |
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/7695/artifact/patchprocess/whitespace.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7695/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/7695/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/7695/console |
This message was automatically generated.
> FairScheduler: Enable delay scheduling by default
> -
>
> Key: YARN-1612
> URL: https://issues.apache.org/jira/browse/YARN-1612
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Reporter: Sandy Ryza
>Assignee: Chen He
> Attachments: YARN-1612-003.patch, YARN-1612-004.patch,
> YARN-1612-v2.patch, YARN-1612.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1612) FairScheduler: Enable delay scheduling by default
[ https://issues.apache.org/jira/browse/YARN-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14527435#comment-14527435 ] Chen He commented on YARN-1612: --- not sure why QA complains the white space. > FairScheduler: Enable delay scheduling by default > - > > Key: YARN-1612 > URL: https://issues.apache.org/jira/browse/YARN-1612 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Reporter: Sandy Ryza >Assignee: Chen He > Attachments: YARN-1612-003.patch, YARN-1612-v2.patch, YARN-1612.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1612) FairScheduler: Enable delay scheduling by default
[
https://issues.apache.org/jira/browse/YARN-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14527226#comment-14527226
]
Hadoop QA commented on YARN-1612:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 41s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 34s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 34s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 49s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 16s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | yarn tests | 52m 40s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 89m 6s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12730242/YARN-1612-003.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / c457876 |
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/7688/artifact/patchprocess/whitespace.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7688/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/7688/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/7688/console |
This message was automatically generated.
> FairScheduler: Enable delay scheduling by default
> -
>
> Key: YARN-1612
> URL: https://issues.apache.org/jira/browse/YARN-1612
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Reporter: Sandy Ryza
>Assignee: Chen He
> Attachments: YARN-1612-003.patch, YARN-1612-v2.patch, YARN-1612.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1612) FairScheduler: Enable delay scheduling by default
[ https://issues.apache.org/jira/browse/YARN-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14526296#comment-14526296 ] Chen He commented on YARN-1612: --- Thank you for reviewing, Karthik, I will update the patch tomorrow. > FairScheduler: Enable delay scheduling by default > - > > Key: YARN-1612 > URL: https://issues.apache.org/jira/browse/YARN-1612 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Reporter: Sandy Ryza >Assignee: Chen He > Attachments: YARN-1612-v2.patch, YARN-1612.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1612) FairScheduler: Enable delay scheduling by default
[ https://issues.apache.org/jira/browse/YARN-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14526232#comment-14526232 ] Karthik Kambatla commented on YARN-1612: Thanks for working on this, Chen. It might be better to make these changes in FairSchedulerConfiguration itself, so the caller (FairScheduler class) is unaware of what the default is. > FairScheduler: Enable delay scheduling by default > - > > Key: YARN-1612 > URL: https://issues.apache.org/jira/browse/YARN-1612 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Reporter: Sandy Ryza >Assignee: Chen He > Attachments: YARN-1612-v2.patch, YARN-1612.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
