[jira] [Commented] (YARN-6706) Refactor ContainerScheduler to make oversubscription change easier
[ https://issues.apache.org/jira/browse/YARN-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090625#comment-16090625 ] Hudson commented on YARN-6706: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12022 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/12022/]) YARN-6706. Refactor ContainerScheduler to make oversubscription change (arun suresh: rev 5b007921cdf01ecc8ed97c164b7d327b8304c529) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManagerRecovery.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/TestContainerSchedulerQueuing.java > Refactor ContainerScheduler to make oversubscription change easier > -- > > Key: YARN-6706 > URL: https://issues.apache.org/jira/browse/YARN-6706 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > Fix For: 3.0.0-beta1 > > Attachments: YARN-6706.01.patch, YARN-6706-YARN-1011.00.patch, > YARN-6706-YARN-1011.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6706) Refactor ContainerScheduler to make oversubscription change easier
[ https://issues.apache.org/jira/browse/YARN-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090593#comment-16090593 ] Haibo Chen commented on YARN-6706: -- Can you also cherry-pick this into YARN-1011 branch? > Refactor ContainerScheduler to make oversubscription change easier > -- > > Key: YARN-6706 > URL: https://issues.apache.org/jira/browse/YARN-6706 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > Fix For: 3.0.0-beta1 > > Attachments: YARN-6706.01.patch, YARN-6706-YARN-1011.00.patch, > YARN-6706-YARN-1011.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6706) Refactor ContainerScheduler to make oversubscription change easier
[ https://issues.apache.org/jira/browse/YARN-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090587#comment-16090587 ] Haibo Chen commented on YARN-6706: -- Thanks [~asuresh] for the reviews and commit. > Refactor ContainerScheduler to make oversubscription change easier > -- > > Key: YARN-6706 > URL: https://issues.apache.org/jira/browse/YARN-6706 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > Fix For: 3.0.0-beta1 > > Attachments: YARN-6706.01.patch, YARN-6706-YARN-1011.00.patch, > YARN-6706-YARN-1011.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6706) Refactor ContainerScheduler to make oversubscription change easier
[ https://issues.apache.org/jira/browse/YARN-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090580#comment-16090580 ] Arun Suresh commented on YARN-6706: --- [~kkaranasos].. Apologies, but think you send the message just as I was committing !! As [~haibochen] mentioned, maybe you can post comments to YARN-6831 and we can incorporate the changes there.. > Refactor ContainerScheduler to make oversubscription change easier > -- > > Key: YARN-6706 > URL: https://issues.apache.org/jira/browse/YARN-6706 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > Fix For: 3.0.0-beta1 > > Attachments: YARN-6706.01.patch, YARN-6706-YARN-1011.00.patch, > YARN-6706-YARN-1011.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6706) Refactor ContainerScheduler to make oversubscription change easier
[ https://issues.apache.org/jira/browse/YARN-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090573#comment-16090573 ] Haibo Chen commented on YARN-6706: -- Thanks [~kkaranasos] for your coming review! FYI, I have created YARN-6831 to address Karthik's comments since it is not totally necessary to have them in this jira. > Refactor ContainerScheduler to make oversubscription change easier > -- > > Key: YARN-6706 > URL: https://issues.apache.org/jira/browse/YARN-6706 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-6706.01.patch, YARN-6706-YARN-1011.00.patch, > YARN-6706-YARN-1011.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6706) Refactor ContainerScheduler to make oversubscription change easier
[ https://issues.apache.org/jira/browse/YARN-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090555#comment-16090555 ] Konstantinos Karanasos commented on YARN-6706: -- Hi guys, I am back. If it is possible, please wait one more day so that I can give it a look as well. Thanks! -- Konstantinos > Refactor ContainerScheduler to make oversubscription change easier > -- > > Key: YARN-6706 > URL: https://issues.apache.org/jira/browse/YARN-6706 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-6706.01.patch, YARN-6706-YARN-1011.00.patch, > YARN-6706-YARN-1011.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6706) Refactor ContainerScheduler to make oversubscription change easier
[ https://issues.apache.org/jira/browse/YARN-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090544#comment-16090544 ] Arun Suresh commented on YARN-6706: --- Thanks Haibo.. +1 for the latest patch.. will check it in shortly > Refactor ContainerScheduler to make oversubscription change easier > -- > > Key: YARN-6706 > URL: https://issues.apache.org/jira/browse/YARN-6706 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-6706.01.patch, YARN-6706-YARN-1011.00.patch, > YARN-6706-YARN-1011.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6706) Refactor ContainerScheduler to make oversubscription change easier
[ https://issues.apache.org/jira/browse/YARN-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084106#comment-16084106 ] Haibo Chen commented on YARN-6706: -- Filed YARN-6812 to fix remove maxOppQueueLength. Any more comment? > Refactor ContainerScheduler to make oversubscription change easier > -- > > Key: YARN-6706 > URL: https://issues.apache.org/jira/browse/YARN-6706 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-6706.01.patch, YARN-6706-YARN-1011.00.patch, > YARN-6706-YARN-1011.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6706) Refactor ContainerScheduler to make oversubscription change easier
[ https://issues.apache.org/jira/browse/YARN-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16083931#comment-16083931 ] Arun Suresh commented on YARN-6706: --- Agreed... lets discuss this in a follow up JIRA. > Refactor ContainerScheduler to make oversubscription change easier > -- > > Key: YARN-6706 > URL: https://issues.apache.org/jira/browse/YARN-6706 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-6706.01.patch, YARN-6706-YARN-1011.00.patch, > YARN-6706-YARN-1011.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6706) Refactor ContainerScheduler to make oversubscription change easier
[ https://issues.apache.org/jira/browse/YARN-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16083467#comment-16083467 ] Haibo Chen commented on YARN-6706: -- hmm... Looks like the maxOppQueueLength is also used in MiniYarnCluster, which is used in many tests. Probably need more time to check whether there is some impact. Do you mind if I fix it in a follow up jira? Once this is committed, we can unblock YARN-6675 and others. In the meantime, I can upload another patch to fix maxOppQueueLength. > Refactor ContainerScheduler to make oversubscription change easier > -- > > Key: YARN-6706 > URL: https://issues.apache.org/jira/browse/YARN-6706 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-6706.01.patch, YARN-6706-YARN-1011.00.patch, > YARN-6706-YARN-1011.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6706) Refactor ContainerScheduler to make oversubscription change easier
[ https://issues.apache.org/jira/browse/YARN-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16083352#comment-16083352 ] Haibo Chen commented on YARN-6706: -- Thanks for your reviews! I'll update the patch to get rid of maxOppQueueLength. > Refactor ContainerScheduler to make oversubscription change easier > -- > > Key: YARN-6706 > URL: https://issues.apache.org/jira/browse/YARN-6706 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-6706.01.patch, YARN-6706-YARN-1011.00.patch, > YARN-6706-YARN-1011.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6706) Refactor ContainerScheduler to make oversubscription change easier
[ https://issues.apache.org/jira/browse/YARN-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16083261#comment-16083261 ] Arun Suresh commented on YARN-6706: --- bq. You are referring to the else statement in ContainerScheduler#scheduleContainer() right? The reason of doing startPendingContainers() both before and after en-queuing is so that we always respect the max queue limit for OPPORTUNISTIC containers. Yup.. makes sense.. But was thinking about Karthik's [comment|https://issues.apache.org/jira/browse/YARN-6706?focusedCommentId=16080813=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16080813] and maybe we can get rid of *maxOppQueueLength*. I prefer just using the *queuingLimit*.. It also reduces configuration hassle, since *queuingLimit* is calculated centrally by the RM. The only reason we might need to have a locally configured maxOppQueueLength is in the case of heterogeneous clusters where max number of queued containers should differ from NM to NM - but to be honest, I am not sure it would be useful. > Refactor ContainerScheduler to make oversubscription change easier > -- > > Key: YARN-6706 > URL: https://issues.apache.org/jira/browse/YARN-6706 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-6706.01.patch, YARN-6706-YARN-1011.00.patch, > YARN-6706-YARN-1011.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6706) Refactor ContainerScheduler to make oversubscription change easier
[ https://issues.apache.org/jira/browse/YARN-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16081275#comment-16081275 ] Haibo Chen commented on YARN-6706: -- bq. Not sure if startPendingContainers() before and after en-queuing is usefull You are referring to the else statement in ContainerScheduler#scheduleContainer() right? The reason of doing startPendingContainers() both before and after en-queuing is so that we always respect the max queue limit for OPPORTUNISTIC containers. If we were to always do enqueueing first and then startPendingContainers(), we could end up going over the OPPR container queue length. For GUARANTEED containers, killOpportunisticContainers is needed if the GUARANTEED container stays in the queue after startPendingContainers(). If I misunderstood your comment, can you elaborate a little more. > Refactor ContainerScheduler to make oversubscription change easier > -- > > Key: YARN-6706 > URL: https://issues.apache.org/jira/browse/YARN-6706 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-6706.01.patch, YARN-6706-YARN-1011.00.patch, > YARN-6706-YARN-1011.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6706) Refactor ContainerScheduler to make oversubscription change easier
[
https://issues.apache.org/jira/browse/YARN-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16081132#comment-16081132
]
Arun Suresh commented on YARN-6706:
---
Yup. Agree this can go into trunk.
The patch generally looks good. One comment:
In the {{ContainerScheduler#scheduleContainer()}}, I see how
startPendingContainers() will ensure that we do not have to wait for a
completed container to start containers, but it maybe be better to just move
all the {{startPendingContainers()}} invocations from inside the if..then..else
to the last line of the method. Not sure if startPendingContainers() before and
after en-queuing is usefull.
> Refactor ContainerScheduler to make oversubscription change easier
> --
>
> Key: YARN-6706
> URL: https://issues.apache.org/jira/browse/YARN-6706
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha3
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: YARN-6706.01.patch, YARN-6706-YARN-1011.00.patch,
> YARN-6706-YARN-1011.01.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6706) Refactor ContainerScheduler to make oversubscription change easier
[
https://issues.apache.org/jira/browse/YARN-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16080916#comment-16080916
]
Hadoop QA commented on YARN-6706:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
46s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 5 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 15m
8s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6706 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12876466/YARN-6706.01.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux dc1855100dc5 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 09653ea |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/16352/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/16352/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/16352/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Refactor ContainerScheduler to make oversubscription change easier
> --
>
> Key: YARN-6706
> URL: https://issues.apache.org/jira/browse/YARN-6706
> Project:
[jira] [Commented] (YARN-6706) Refactor ContainerScheduler to make oversubscription change easier
[ https://issues.apache.org/jira/browse/YARN-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16080835#comment-16080835 ] Haibo Chen commented on YARN-6706: -- Would appreciate reviews from [~arun.sur...@gmail.com] and [~kkaranasos] to see if the changes look OK to you folks. > Refactor ContainerScheduler to make oversubscription change easier > -- > > Key: YARN-6706 > URL: https://issues.apache.org/jira/browse/YARN-6706 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-6706.01.patch, YARN-6706-YARN-1011.00.patch, > YARN-6706-YARN-1011.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6706) Refactor ContainerScheduler to make oversubscription change easier
[ https://issues.apache.org/jira/browse/YARN-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16080813#comment-16080813 ] Karthik Kambatla commented on YARN-6706: I feel we should get this into trunk directly instead of YARN-1011 branch. The current patch looks okay, Can we add a few more things: # Make ResourceUtilizationTracker pluggable. That way, we could use a different tracker when oversubscription is enabled. # ContainerScheduler ## Why do we need maxOppQueueLength given queuingLimit? ## Is there value in splitting runningContainers into runningGuaranteed and runningOpportunistic? ## getOpportunisticContainersStatus method implementation feels awkward. How about capturing the state in the field here, and have metrics etc. pull from here? ## startContainersFromQueue: Local variable resourcesAvailable is unnecessary # OpportunisticContainersStatus ## Let us clearly differentiate between allocated, used and utilized. Maybe, we should rename current *Used* methods to *Allocated*? ## I prefer either full name Opportunistic (in method) or Opp (shortest name that makes sense). Opport is neither short nor fully descriptive. ## Have we considered folding ContainerQueuingLimit class into this? /cc [~asuresh] and [~kkaranasos] > Refactor ContainerScheduler to make oversubscription change easier > -- > > Key: YARN-6706 > URL: https://issues.apache.org/jira/browse/YARN-6706 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-6706-YARN-1011.00.patch, > YARN-6706-YARN-1011.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6706) Refactor ContainerScheduler to make oversubscription change easier
[
https://issues.apache.org/jira/browse/YARN-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054510#comment-16054510
]
Hadoop QA commented on YARN-6706:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
18s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} YARN-1011 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
51s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in YARN-1011 has 5 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
28s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6706 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12873526/YARN-6706-YARN-1011.01.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 783c0dcc012b 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-1011 / 153498b |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/16201/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/16201/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/16201/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Refactor ContainerScheduler to make oversubscription change easier
> --
>
> Key: YARN-6706
> URL: https://issues.apache.org/jira/browse/YARN-6706
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha3
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments:
[jira] [Commented] (YARN-6706) Refactor ContainerScheduler to make oversubscription change easier
[ https://issues.apache.org/jira/browse/YARN-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054479#comment-16054479 ] Haibo Chen commented on YARN-6706: -- [~leftnoteasy] I have shared the latest design doc with you. > Refactor ContainerScheduler to make oversubscription change easier > -- > > Key: YARN-6706 > URL: https://issues.apache.org/jira/browse/YARN-6706 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-6706-YARN-1011.00.patch, > YARN-6706-YARN-1011.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6706) Refactor ContainerScheduler to make oversubscription change easier
[ https://issues.apache.org/jira/browse/YARN-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054402#comment-16054402 ] Wangda Tan commented on YARN-6706: -- [~haibo.chen], [~asuresh], [~kkaranasos], is there any design doc to understand what is the overall scope of the oversubscription feature? Is the design doc attached to YARN-1011 is up-to-dated? Thanks, > Refactor ContainerScheduler to make oversubscription change easier > -- > > Key: YARN-6706 > URL: https://issues.apache.org/jira/browse/YARN-6706 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-6706-YARN-1011.00.patch, > YARN-6706-YARN-1011.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6706) Refactor ContainerScheduler to make oversubscription change easier
[ https://issues.apache.org/jira/browse/YARN-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054388#comment-16054388 ] Haibo Chen commented on YARN-6706: -- Patch updated to address checkstyle and unit test issues. The findbugs warnings are unrelated. > Refactor ContainerScheduler to make oversubscription change easier > -- > > Key: YARN-6706 > URL: https://issues.apache.org/jira/browse/YARN-6706 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-6706-YARN-1011.00.patch, > YARN-6706-YARN-1011.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6706) Refactor ContainerScheduler to make oversubscription change easier
[
https://issues.apache.org/jira/browse/YARN-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045262#comment-16045262
]
Hadoop QA commented on YARN-6706:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
32s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} YARN-1011 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
43s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in YARN-1011 has 5 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 14s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 3 new + 1 unchanged - 0 fixed = 4 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 12m 59s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 21s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.containermanager.TestContainerManagerRecovery |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6706 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12872378/YARN-6706-YARN-1011.00.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 54d856200e0f 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-1011 / 153498b |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/16175/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/16175/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/16175/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/16175/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
[jira] [Commented] (YARN-6706) Refactor ContainerScheduler to make oversubscription change easier
[ https://issues.apache.org/jira/browse/YARN-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045233#comment-16045233 ] Haibo Chen commented on YARN-6706: -- [~arun.sur...@gmail.com] [~kkaranasos] please kindly review > Refactor ContainerScheduler to make oversubscription change easier > -- > > Key: YARN-6706 > URL: https://issues.apache.org/jira/browse/YARN-6706 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-6706-YARN-1011.00.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6706) Refactor ContainerScheduler to make oversubscription change easier
[ https://issues.apache.org/jira/browse/YARN-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045229#comment-16045229 ] Haibo Chen commented on YARN-6706: -- This patch does the following: 1) Changes the ContainerScheduler.scheduleContainer() logic if the container is guaranteed, we first put it into the guaranteed container queue, then do a pass-through guaranteed queue followed by opportunistic queue to start as many containers as possible. Finally, if the container remains in the guaranteed queue, we know we need to kill some opportunistic container if the container is opportunistic, we first do a pass through pass-through guaranteed queue followed by opportunistic queue to start as many containers as possible. Then we try to enqueue the opportunistic container, which may fail if the maxOppQueue length is reached. Lastly, we may be able to start it if there happen to be some resources remaining. This way, we slightly change the semantic of maxOppQueueLength in cases where oversubscription is enabled. We enqueue an opportunistic container when neither there is unallocated resource nor can it be started due to oversubscription. 2) When maxOppQUeueLength is set to 0 or a negative value, containers are no longer unconditionally started. 3) Adds a new unit test in TestContainerSchedulerQueuing to test the case where the OppQueue length limit is reached with too many opportunistic container requests. 4) Add a new method resourceAvailableToStartContainer() which we could extend to include oversubscription logic. > Refactor ContainerScheduler to make oversubscription change easier > -- > > Key: YARN-6706 > URL: https://issues.apache.org/jira/browse/YARN-6706 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-6706-YARN-1011.00.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
