[jira] [Commented] (YARN-7610) Extend Distributed Shell to support launching job with opportunistic containers
[
https://issues.apache.org/jira/browse/YARN-7610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279961#comment-16279961
]
Hudson commented on YARN-7610:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13335 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13335/])
YARN-7610. Extend Distributed Shell to support launching job with (wwei: rev
40b0045ebe0752cd3d1d09be00acbabdea983799)
* (add)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/OpportunisticContainers.md.vm
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
* (delete)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/OpportunisticContainers.md
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java
> Extend Distributed Shell to support launching job with opportunistic
> containers
> ---
>
> Key: YARN-7610
> URL: https://issues.apache.org/jira/browse/YARN-7610
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: applications/distributed-shell
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: YARN-7610.001.patch, YARN-7610.002.patch,
> YARN-7610.003.patch, YARN-7610.004.patch, YARN-7610.005.patch, added_doc.png,
> outline_compare.png
>
>
> Per doc in
> [https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html#Running_a_Sample_Job],
> user can run some of PI job mappers as O containers. Similarly, propose to
> extend distributed shell to support specifying the container type, it will be
> very helpful for testing. Propose to add following argument
> {code}
> $./bin/yarn org.apache.hadoop.yarn.applications.distributedshell.Client
> -container_type Container execution type,
> GUARANTEED or
> OPPORTUNISTIC
> {code}
> Implication: all containers in a distributed shell job will be launching as
> user-specified container type (except for AM), if not given, default type is
> {{GUARANTEED}}. AM is always launched as {{GUARANTEED}} container.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7610) Extend Distributed Shell to support launching job with opportunistic containers
[
https://issues.apache.org/jira/browse/YARN-7610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279938#comment-16279938
]
Weiwei Yang commented on YARN-7610:
---
There are still 4 line(s) that end in whitespace in v5 patch, but they are not
introduced by any of the changes in the patch. I tried to fix them but that
would cause the patch won't apply. I think we can just let it as it is now. The
v5 patch should be good, committing now.
> Extend Distributed Shell to support launching job with opportunistic
> containers
> ---
>
> Key: YARN-7610
> URL: https://issues.apache.org/jira/browse/YARN-7610
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: applications/distributed-shell
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: YARN-7610.001.patch, YARN-7610.002.patch,
> YARN-7610.003.patch, YARN-7610.004.patch, YARN-7610.005.patch, added_doc.png,
> outline_compare.png
>
>
> Per doc in
> [https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html#Running_a_Sample_Job],
> user can run some of PI job mappers as O containers. Similarly, propose to
> extend distributed shell to support specifying the container type, it will be
> very helpful for testing. Propose to add following argument
> {code}
> $./bin/yarn org.apache.hadoop.yarn.applications.distributedshell.Client
> -container_type Container execution type,
> GUARANTEED or
> OPPORTUNISTIC
> {code}
> Implication: all containers in a distributed shell job will be launching as
> user-specified container type (except for AM), if not given, default type is
> {{GUARANTEED}}. AM is always launched as {{GUARANTEED}} container.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7610) Extend Distributed Shell to support launching job with opportunistic containers
[
https://issues.apache.org/jira/browse/YARN-7610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279931#comment-16279931
]
genericqa commented on YARN-7610:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
48s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 43s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
1s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 2s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 2 new + 207 unchanged - 0 fixed = 209 total (was 207) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 55s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m
19s{color} | {color:green} hadoop-yarn-applications-distributedshell in the
patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
18s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 75m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7610 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12900826/YARN-7610.005.patch |
| Optional Tests | asflicense compile
[jira] [Commented] (YARN-7610) Extend Distributed Shell to support launching job with opportunistic containers
[
https://issues.apache.org/jira/browse/YARN-7610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279845#comment-16279845
]
genericqa commented on YARN-7610:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 25s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
32s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 58s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 2 new + 207 unchanged - 0 fixed = 209 total (was 207) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 15s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
47s{color} | {color:green} hadoop-yarn-applications-distributedshell in the
patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
20s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 76m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7610 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12900815/YARN-7610.003.patch |
| Optional Tests | asflicense compile
[jira] [Commented] (YARN-7610) Extend Distributed Shell to support launching job with opportunistic containers
[
https://issues.apache.org/jira/browse/YARN-7610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279821#comment-16279821
]
genericqa commented on YARN-7610:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} YARN-7610 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-7610 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12900825/YARN-7610.004.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/18812/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Extend Distributed Shell to support launching job with opportunistic
> containers
> ---
>
> Key: YARN-7610
> URL: https://issues.apache.org/jira/browse/YARN-7610
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: applications/distributed-shell
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: YARN-7610.001.patch, YARN-7610.002.patch,
> YARN-7610.003.patch, YARN-7610.004.patch, added_doc.png, outline_compare.png
>
>
> Per doc in
> [https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html#Running_a_Sample_Job],
> user can run some of PI job mappers as O containers. Similarly, propose to
> extend distributed shell to support specifying the container type, it will be
> very helpful for testing. Propose to add following argument
> {code}
> $./bin/yarn org.apache.hadoop.yarn.applications.distributedshell.Client
> -container_type Container execution type,
> GUARANTEED or
> OPPORTUNISTIC
> {code}
> Implication: all containers in a distributed shell job will be launching as
> user-specified container type (except for AM), if not given, default type is
> {{GUARANTEED}}. AM is always launched as {{GUARANTEED}} container.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7610) Extend Distributed Shell to support launching job with opportunistic containers
[
https://issues.apache.org/jira/browse/YARN-7610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279796#comment-16279796
]
Weiwei Yang commented on YARN-7610:
---
Thanks for the +1 [~asuresh], the patch has whitespace issue, I will fix that
in v4 and once I get a clean jenkins result I will commit this to branch-2.9,
branch-3.0 and trunk. Thanks for the review all along.
> Extend Distributed Shell to support launching job with opportunistic
> containers
> ---
>
> Key: YARN-7610
> URL: https://issues.apache.org/jira/browse/YARN-7610
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: applications/distributed-shell
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: YARN-7610.001.patch, YARN-7610.002.patch,
> YARN-7610.003.patch, added_doc.png, outline_compare.png
>
>
> Per doc in
> [https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html#Running_a_Sample_Job],
> user can run some of PI job mappers as O containers. Similarly, propose to
> extend distributed shell to support specifying the container type, it will be
> very helpful for testing. Propose to add following argument
> {code}
> $./bin/yarn org.apache.hadoop.yarn.applications.distributedshell.Client
> -container_type Container execution type,
> GUARANTEED or
> OPPORTUNISTIC
> {code}
> Implication: all containers in a distributed shell job will be launching as
> user-specified container type (except for AM), if not given, default type is
> {{GUARANTEED}}. AM is always launched as {{GUARANTEED}} container.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7610) Extend Distributed Shell to support launching job with opportunistic containers
[
https://issues.apache.org/jira/browse/YARN-7610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279792#comment-16279792
]
genericqa commented on YARN-7610:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 42s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
34s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 50s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 2 new + 207 unchanged - 0 fixed = 209 total (was 207) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 23s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
9s{color} | {color:green} hadoop-yarn-applications-distributedshell in the
patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
14s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 68m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7610 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12900806/YARN-7610.002.patch |
| Optional Tests | asflicense compile
[jira] [Commented] (YARN-7610) Extend Distributed Shell to support launching job with opportunistic containers
[
https://issues.apache.org/jira/browse/YARN-7610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279786#comment-16279786
]
Arun Suresh commented on YARN-7610:
---
+1 Thanks for the quick update [~cheersyang]
> Extend Distributed Shell to support launching job with opportunistic
> containers
> ---
>
> Key: YARN-7610
> URL: https://issues.apache.org/jira/browse/YARN-7610
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: applications/distributed-shell
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: YARN-7610.001.patch, YARN-7610.002.patch,
> YARN-7610.003.patch, added_doc.png, outline_compare.png
>
>
> Per doc in
> [https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html#Running_a_Sample_Job],
> user can run some of PI job mappers as O containers. Similarly, propose to
> extend distributed shell to support specifying the container type, it will be
> very helpful for testing. Propose to add following argument
> {code}
> $./bin/yarn org.apache.hadoop.yarn.applications.distributedshell.Client
> -container_type Container execution type,
> GUARANTEED or
> OPPORTUNISTIC
> {code}
> Implication: all containers in a distributed shell job will be launching as
> user-specified container type (except for AM), if not given, default type is
> {{GUARANTEED}}. AM is always launched as {{GUARANTEED}} container.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7610) Extend Distributed Shell to support launching job with opportunistic containers
[
https://issues.apache.org/jira/browse/YARN-7610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279771#comment-16279771
]
Weiwei Yang commented on YARN-7610:
---
Hi [~asuresh]
bq. Since you are updating the documentation, can you also include the
following changes.
Sure thing. I assume you mean the changes to {{OpportunisticContainers.md}} in
[this
commit|https://github.com/apache/hadoop/commit/4e1fd7b022362ec7954b23d0ba6420dbc72a031b],
if that's the case, I can add that in v3 patch.
> Extend Distributed Shell to support launching job with opportunistic
> containers
> ---
>
> Key: YARN-7610
> URL: https://issues.apache.org/jira/browse/YARN-7610
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: applications/distributed-shell
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: YARN-7610.001.patch, YARN-7610.002.patch, added_doc.png,
> outline_compare.png
>
>
> Per doc in
> [https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html#Running_a_Sample_Job],
> user can run some of PI job mappers as O containers. Similarly, propose to
> extend distributed shell to support specifying the container type, it will be
> very helpful for testing. Propose to add following argument
> {code}
> $./bin/yarn org.apache.hadoop.yarn.applications.distributedshell.Client
> -container_type Container execution type,
> GUARANTEED or
> OPPORTUNISTIC
> {code}
> Implication: all containers in a distributed shell job will be launching as
> user-specified container type (except for AM), if not given, default type is
> {{GUARANTEED}}. AM is always launched as {{GUARANTEED}} container.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7610) Extend Distributed Shell to support launching job with opportunistic containers
[
https://issues.apache.org/jira/browse/YARN-7610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279724#comment-16279724
]
Arun Suresh commented on YARN-7610:
---
Thanks for the update [~cheersyang]..
Since you are updating the OpportunisticContainers.md.vm file, can you also
include the following
[changes|https://github.com/apache/hadoop/commit/4e1fd7b022362ec7954b23d0ba6420dbc72a031b].
I had put them in when we released 2.9.0, but forgot to cherry-pick to
branch-2.9 and above.
Since we are modifying the md to an md.vm, Can you also run mvn site and post a
png of the updated section of the generated html page here. so we can verify if
it looks good ?
> Extend Distributed Shell to support launching job with opportunistic
> containers
> ---
>
> Key: YARN-7610
> URL: https://issues.apache.org/jira/browse/YARN-7610
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: applications/distributed-shell
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: YARN-7610.001.patch, YARN-7610.002.patch
>
>
> Per doc in
> [https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html#Running_a_Sample_Job],
> user can run some of PI job mappers as O containers. Similarly, propose to
> extend distributed shell to support specifying the container type, it will be
> very helpful for testing. Propose to add following argument
> {code}
> $./bin/yarn org.apache.hadoop.yarn.applications.distributedshell.Client
> -container_type Container execution type,
> GUARANTEED or
> OPPORTUNISTIC
> {code}
> Implication: all containers in a distributed shell job will be launching as
> user-specified container type (except for AM), if not given, default type is
> {{GUARANTEED}}. AM is always launched as {{GUARANTEED}} container.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7610) Extend Distributed Shell to support launching job with opportunistic containers
[
https://issues.apache.org/jira/browse/YARN-7610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279710#comment-16279710
]
Weiwei Yang commented on YARN-7610:
---
I found {{OpportunisticContainers.md}} was using hardcoded version name like
following
{noformat}
hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-alpha2-SNAPSHOT.jar
{noformat}
I will fix that as well in next patch.
> Extend Distributed Shell to support launching job with opportunistic
> containers
> ---
>
> Key: YARN-7610
> URL: https://issues.apache.org/jira/browse/YARN-7610
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: applications/distributed-shell
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: YARN-7610.001.patch
>
>
> Per doc in
> [https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html#Running_a_Sample_Job],
> user can run some of PI job mappers as O containers. Similarly, propose to
> extend distributed shell to support specifying the container type, it will be
> very helpful for testing. Propose to add following argument
> {code}
> $./bin/yarn org.apache.hadoop.yarn.applications.distributedshell.Client
> -container_type Container execution type,
> GUARANTEED or
> OPPORTUNISTIC
> {code}
> Implication: all containers in a distributed shell job will be launching as
> user-specified container type (except for AM), if not given, default type is
> {{GUARANTEED}}. AM is always launched as {{GUARANTEED}} container.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7610) Extend Distributed Shell to support launching job with opportunistic containers
[
https://issues.apache.org/jira/browse/YARN-7610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279602#comment-16279602
]
Arun Suresh commented on YARN-7610:
---
Makes sense - we can track that in another JIRA
> Extend Distributed Shell to support launching job with opportunistic
> containers
> ---
>
> Key: YARN-7610
> URL: https://issues.apache.org/jira/browse/YARN-7610
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: applications/distributed-shell
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: YARN-7610.001.patch
>
>
> Per doc in
> [https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html#Running_a_Sample_Job],
> user can run some of PI job mappers as O containers. Similarly, propose to
> extend distributed shell to support specifying the container type, it will be
> very helpful for testing. Propose to add following argument
> {code}
> $./bin/yarn org.apache.hadoop.yarn.applications.distributedshell.Client
> -container_type Container execution type,
> GUARANTEED or
> OPPORTUNISTIC
> {code}
> Implication: all containers in a distributed shell job will be launching as
> user-specified container type (except for AM), if not given, default type is
> {{GUARANTEED}}. AM is always launched as {{GUARANTEED}} container.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7610) Extend Distributed Shell to support launching job with opportunistic containers
[
https://issues.apache.org/jira/browse/YARN-7610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279585#comment-16279585
]
Weiwei Yang commented on YARN-7610:
---
Hi [~asuresh]
bq. Can you also update the doc as well ?
Sure, I will add some doc to {{OpportunisticContainers.md}} in next patch.
bq. If the flag is enabled, the distributed shell AM will first ask for opp
containers like you have done in the patch, and once it starts, the AM will
send an update request to the RM to promote the containers to guaranteed.
I am perfectly OK with this, by adding another tag, e.g
{{container_prompt_on_start=true}}, default value is false. But if possible can
we track this in another JIRA as it is less important, I want to get this
committed ASAP so rest of my team member can easily try out.
Please let me know your thoughts, thanks.
> Extend Distributed Shell to support launching job with opportunistic
> containers
> ---
>
> Key: YARN-7610
> URL: https://issues.apache.org/jira/browse/YARN-7610
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: applications/distributed-shell
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: YARN-7610.001.patch
>
>
> Per doc in
> [https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html#Running_a_Sample_Job],
> user can run some of PI job mappers as O containers. Similarly, propose to
> extend distributed shell to support specifying the container type, it will be
> very helpful for testing. Propose to add following argument
> {code}
> $./bin/yarn org.apache.hadoop.yarn.applications.distributedshell.Client
> -container_type Container execution type,
> GUARANTEED or
> OPPORTUNISTIC
> {code}
> Implication: all containers in a distributed shell job will be launching as
> user-specified container type (except for AM), if not given, default type is
> {{GUARANTEED}}. AM is always launched as {{GUARANTEED}} container.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7610) Extend Distributed Shell to support launching job with opportunistic containers
[
https://issues.apache.org/jira/browse/YARN-7610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16278971#comment-16278971
]
Arun Suresh commented on YARN-7610:
---
Thanks for taking a stab at this [~cheersyang].. Much appreciated!!
Couple of comments:
* Can you also update the doc as well ?
* I was also hoping to demonstrate container promotion as well. I was thinking
to add a flag to promote opportunistic containers. If the flag is enabled, the
distributed shell AM will first ask for opp containers like you have done in
the patch, and once it starts, the AM will send an update request to the RM to
promote the containers to guaranteed.
> Extend Distributed Shell to support launching job with opportunistic
> containers
> ---
>
> Key: YARN-7610
> URL: https://issues.apache.org/jira/browse/YARN-7610
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: applications/distributed-shell
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: YARN-7610.001.patch
>
>
> Per doc in
> [https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html#Running_a_Sample_Job],
> user can run some of PI job mappers as O containers. Similarly, propose to
> extend distributed shell to support specifying the container type, it will be
> very helpful for testing. Propose to add following argument
> {code}
> $./bin/yarn org.apache.hadoop.yarn.applications.distributedshell.Client
> -container_type Container execution type,
> GUARANTEED or
> OPPORTUNISTIC
> {code}
> Implication: all containers in a distributed shell job will be launching as
> user-specified container type (except for AM), if not given, default type is
> {{GUARANTEED}}. AM is always launched as {{GUARANTEED}} container.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7610) Extend Distributed Shell to support launching job with opportunistic containers
[
https://issues.apache.org/jira/browse/YARN-7610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16278415#comment-16278415
]
Weiwei Yang commented on YARN-7610:
---
Hi [~kkaranasos]
Could you please help to review? Checkstyle issues were complaining about the
init method exceeds 150 lines, which is not introduced by this patch. Thanks.
> Extend Distributed Shell to support launching job with opportunistic
> containers
> ---
>
> Key: YARN-7610
> URL: https://issues.apache.org/jira/browse/YARN-7610
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: applications/distributed-shell
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: YARN-7610.001.patch
>
>
> Per doc in
> [https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html#Running_a_Sample_Job],
> user can run some of PI job mappers as O containers. Similarly, propose to
> extend distributed shell to support specifying the container type, it will be
> very helpful for testing. Propose to add following argument
> {code}
> $./bin/yarn org.apache.hadoop.yarn.applications.distributedshell.Client
> -container_type Container execution type,
> GUARANTEED or
> OPPORTUNISTIC
> {code}
> Implication: all containers in a distributed shell job will be launching as
> user-specified container type (except for AM), if not given, default type is
> {{GUARANTEED}}. AM is always launched as {{GUARANTEED}} container.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7610) Extend Distributed Shell to support launching job with opportunistic containers
[
https://issues.apache.org/jira/browse/YARN-7610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16278341#comment-16278341
]
genericqa commented on YARN-7610:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 1s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 14s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell:
The patch generated 2 new + 207 unchanged - 0 fixed = 209 total (was 207)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 28s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
37s{color} | {color:green} hadoop-yarn-applications-distributedshell in the
patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7610 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12900641/YARN-7610.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 41e8ae6827df 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 9f1bdaf |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/18787/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/18787/testReport/ |
| Max. process+thread count | 634 (vs. ulimit of 5000) |
| modules | C:
