[jira] [Commented] (YARN-7491) Make sure AM is not scheduled on an opportunistic container
[ https://issues.apache.org/jira/browse/YARN-7491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269506#comment-16269506 ] Haibo Chen commented on YARN-7491: -- Thanks [~miklos.szeg...@cloudera.com] for the reviews! > Make sure AM is not scheduled on an opportunistic container > --- > > Key: YARN-7491 > URL: https://issues.apache.org/jira/browse/YARN-7491 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-7491-YARN-1011.00.patch, > YARN-7491-YARN-1011.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7491) Make sure AM is not scheduled on an opportunistic container
[ https://issues.apache.org/jira/browse/YARN-7491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269490#comment-16269490 ] Miklos Szegedi commented on YARN-7491: -- +1 Thank you for the contribution [~haibochen]. Committing this shortly. > Make sure AM is not scheduled on an opportunistic container > --- > > Key: YARN-7491 > URL: https://issues.apache.org/jira/browse/YARN-7491 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-7491-YARN-1011.00.patch, > YARN-7491-YARN-1011.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7491) Make sure AM is not scheduled on an opportunistic container
[
https://issues.apache.org/jira/browse/YARN-7491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269479#comment-16269479
]
genericqa commented on YARN-7491:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} YARN-1011 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 3m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
39s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
5s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 22s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
59s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
in YARN-1011 has 2 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} YARN-1011 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 27s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
50s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 55m
40s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}110m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7491 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12899667/YARN-7491-YARN-1011.01.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 11994d8484c2 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | YARN-1011 / 10aad13 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
|
[jira] [Commented] (YARN-7491) Make sure AM is not scheduled on an opportunistic container
[ https://issues.apache.org/jira/browse/YARN-7491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269330#comment-16269330 ] Haibo Chen commented on YARN-7491: -- Thanks [~miklos.szeg...@cloudera.com] for the comments. I have updated the patch accordingly. > Make sure AM is not scheduled on an opportunistic container > --- > > Key: YARN-7491 > URL: https://issues.apache.org/jira/browse/YARN-7491 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-7491-YARN-1011.00.patch, > YARN-7491-YARN-1011.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7491) Make sure AM is not scheduled on an opportunistic container
[ https://issues.apache.org/jira/browse/YARN-7491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269285#comment-16269285 ] Miklos Szegedi commented on YARN-7491: -- We discussed this in person. So the issue is that there is no unit test newAMResourceRequest and probably it needs to be based on newResourceRequest to group the common code together. Also, please address the checkstyle issue. > Make sure AM is not scheduled on an opportunistic container > --- > > Key: YARN-7491 > URL: https://issues.apache.org/jira/browse/YARN-7491 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-7491-YARN-1011.00.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7491) Make sure AM is not scheduled on an opportunistic container
[
https://issues.apache.org/jira/browse/YARN-7491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268065#comment-16268065
]
Haibo Chen commented on YARN-7491:
--
That's because of this
{code}
@@ -491,7 +491,7 @@ private RMAppImpl createAndPopulateNewRMApp(
for (ResourceRequest amReq : amReqs) {
amReq.setCapability(anyReq.getCapability());
amReq.setExecutionTypeRequest(
- ExecutionTypeRequest.newInstance(ExecutionType.GUARANTEED));
+ ExecutionTypeRequest.newInstance(ExecutionType.GUARANTEED,
true));
amReq.setNumContainers(1);
amReq.setPriority(RMAppAttemptImpl.AM_CONTAINER_PRIORITY);
}
{code}
which is the core of this patch.
All AM resource requests, whether specified in submission context or not, will
be normalized as above.
If you revert that core change, TestAppManager will fail.
Though not absolutely necessary,
{code}
if (amReqs == null || amReqs.isEmpty()) {
if (submissionContext.getResource() != null) {
amReqs = Collections.singletonList(BuilderUtils
- .newResourceRequest(RMAppAttemptImpl.AM_CONTAINER_PRIORITY,
- ResourceRequest.ANY, submissionContext.getResource(), 1));
+ .newAMResourceRequest(RMAppAttemptImpl.AM_CONTAINER_PRIORITY,
+ ResourceRequest.ANY, submissionContext.getResource()));
{code}
makes it very clear that we always set the enforceExecutionType to true for AM
resource requests.
> Make sure AM is not scheduled on an opportunistic container
> ---
>
> Key: YARN-7491
> URL: https://issues.apache.org/jira/browse/YARN-7491
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: YARN-7491-YARN-1011.00.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7491) Make sure AM is not scheduled on an opportunistic container
[
https://issues.apache.org/jira/browse/YARN-7491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267920#comment-16267920
]
Miklos Szegedi commented on YARN-7491:
--
I reverted the AM request line in the patch to the following:
{code}
amReqs = Collections.singletonList(BuilderUtils
.newResourceRequest(RMAppAttemptImpl.AM_CONTAINER_PRIORITY,
ResourceRequest.ANY, submissionContext.getResource(), 1));
{code}
TestAppManager still succeeded. That means that the change is not covered by
the unit tests to me.
> Make sure AM is not scheduled on an opportunistic container
> ---
>
> Key: YARN-7491
> URL: https://issues.apache.org/jira/browse/YARN-7491
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: YARN-7491-YARN-1011.00.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7491) Make sure AM is not scheduled on an opportunistic container
[ https://issues.apache.org/jira/browse/YARN-7491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267819#comment-16267819 ] Haibo Chen commented on YARN-7491: -- Thanks for the review [~miklos.szeg...@cloudera.com]! I think it is already covered in the unit tests that are changed in the patch. Specifically, we check in the existing unit tests that the ResourceRequests generated for AM containers are always guaranteed with enforceExecutionType set to true. This is also why YARN-7531 is a dependency. > Make sure AM is not scheduled on an opportunistic container > --- > > Key: YARN-7491 > URL: https://issues.apache.org/jira/browse/YARN-7491 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-7491-YARN-1011.00.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7491) Make sure AM is not scheduled on an opportunistic container
[ https://issues.apache.org/jira/browse/YARN-7491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267702#comment-16267702 ] Miklos Szegedi commented on YARN-7491: -- Thank you for the patch [~haibochen]. Even though there are changes in unit tests, so Jenkins does not complain, I think we should have a unit test for the change. > Make sure AM is not scheduled on an opportunistic container > --- > > Key: YARN-7491 > URL: https://issues.apache.org/jira/browse/YARN-7491 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-7491-YARN-1011.00.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7491) Make sure AM is not scheduled on an opportunistic container
[
https://issues.apache.org/jira/browse/YARN-7491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263447#comment-16263447
]
Hadoop QA commented on YARN-7491:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} YARN-1011 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 3m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
37s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
3s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 59s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
0s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
in YARN-1011 has 2 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} YARN-1011 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 35s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 1 new +
111 unchanged - 0 fixed = 112 total (was 111) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 22s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
43s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 59m 20s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}114m 51s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA |
| |
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA |
| | org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7491 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12898936/YARN-7491-YARN-1011.00.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
