[jira] [Updated] (YARN-9077) [Submarine] Improve job state check
[ https://issues.apache.org/jira/browse/YARN-9077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated YARN-9077: --- Attachment: YARN-9077-trunk.002.patch > [Submarine] Improve job state check > --- > > Key: YARN-9077 > URL: https://issues.apache.org/jira/browse/YARN-9077 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Attachments: YARN-9077-trunk.001.patch, YARN-9077-trunk.002.patch > > > Since YARN-8488 gets committed, the service state has "SUCCEEDED" statue. We > should check it instead of "STOPPED". -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9077) [Submarine] Improve job state check
[
https://issues.apache.org/jira/browse/YARN-9077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16706149#comment-16706149
]
Hadoop QA commented on YARN-9077:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 43s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 4s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
32s{color} | {color:green} hadoop-yarn-submarine in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 48m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9077 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12950304/YARN-9077-trunk.002.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 5b60ca4aa209 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8f3e12f |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22766/testReport/ |
| Max. process+thread count | 445 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/22766/console |
| Powered by | Apache Yetus 0.8.0
[jira] [Commented] (YARN-9078) [Submarine] Clean up the code of CliUtils#parseResourcesString
[
https://issues.apache.org/jira/browse/YARN-9078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16706139#comment-16706139
]
Hadoop QA commented on YARN-9078:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 17s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine:
The patch generated 0 new + 5 unchanged - 1 fixed = 5 total (was 6) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 36s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
34s{color} | {color:green} hadoop-yarn-submarine in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9078 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12950303/YARN-9078-trunk.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux e7dc0cb50561 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8f3e12f |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22765/testReport/ |
| Max. process+thread count | 306 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine |
|
[jira] [Commented] (YARN-8870) [Submarine] Add submarine installation scripts
[
https://issues.apache.org/jira/browse/YARN-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16706283#comment-16706283
]
Hadoop QA commented on YARN-8870:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 41s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} shellcheck {color} | {color:red} 0m
3s{color} | {color:red} The patch generated 10 new + 164 unchanged - 35 fixed =
174 total (was 199) {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
17s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 47s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
26s{color} | {color:green} hadoop-yarn-submarine in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-8870 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12950310/YARN-8870.011.patch |
| Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs |
| uname | Linux f2188f1b3196 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8f3e12f |
| maven | version: Apache Maven 3.3.9 |
| shellcheck | v0.4.6 |
| shellcheck |
https://builds.apache.org/job/PreCommit-YARN-Build/22767/artifact/out/diff-patch-shellcheck.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22767/testReport/ |
| Max. process+thread count | 306 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/22767/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> [Submarine] Add submarine installation scripts
> --
>
> Key: YARN-8870
> URL: https://issues.apache.org/jira/browse/YARN-8870
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xun Liu
>Assignee: Xun Liu
>Priority: Critical
> Fix For: 3.2.0
>
> Attachments: YARN-8870-addendum.008.patch, YARN-8870.001.patch,
> YARN-8870.004.patch, YARN-8870.005.patch, YARN-8870.006.patch,
> YARN-8870.007.patch, YARN-8870.009.patch, YARN-8870.010.patch,
> YARN-8870.011.patch
>
>
> In order to reduce the deployment difficulty of Hadoop
> {Submarine} DNS, Docker, GPU, Network, graphics card, operating
[jira] [Updated] (YARN-8870) [Submarine] Add submarine installation scripts
[
https://issues.apache.org/jira/browse/YARN-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xun Liu updated YARN-8870:
--
Attachment: YARN-8870.011.patch
> [Submarine] Add submarine installation scripts
> --
>
> Key: YARN-8870
> URL: https://issues.apache.org/jira/browse/YARN-8870
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xun Liu
>Assignee: Xun Liu
>Priority: Critical
> Fix For: 3.2.0
>
> Attachments: YARN-8870-addendum.008.patch, YARN-8870.001.patch,
> YARN-8870.004.patch, YARN-8870.005.patch, YARN-8870.006.patch,
> YARN-8870.007.patch, YARN-8870.009.patch, YARN-8870.010.patch,
> YARN-8870.011.patch
>
>
> In order to reduce the deployment difficulty of Hadoop
> {Submarine} DNS, Docker, GPU, Network, graphics card, operating system kernel
> modification and other components, I specially developed this installation
> script to deploy Hadoop \{Submarine}
> runtime environment, providing one-click installation Scripts, which can also
> be used to install, uninstall, start, and stop individual components step by
> step.
>
> {color:#ff}design d{color}{color:#FF}ocument：{color}
> [https://docs.google.com/document/d/1muCTGFuUXUvM4JaDYjKqX5liQEg-AsNgkxfLMIFxYHU/edit?usp=sharing]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6523) Newly retrieved security Tokens are sent as part of each heartbeat to each node from RM which is not desirable in large cluster
[ https://issues.apache.org/jira/browse/YARN-6523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16706418#comment-16706418 ] Manikandan R commented on YARN-6523: Thanks for your guidance. Made the changes in NodeHeartbeatResponsePBImpl class to contain only systemCredentialsForAppsProto list and introduced methods in utility class for "Map to/from collection" conversions. With this approach, I can see a minor issue: ApplicationId proto and ByteBuffer conversion happens at the collar side and can repeat for the same heart beat response. Whereas, before, thats not the case. Hope this is not an critical issue. Also created 2 seperate test cases to test token renewal -> sequence no increment flow and sequence no change -> response contains credentials flow. > Newly retrieved security Tokens are sent as part of each heartbeat to each > node from RM which is not desirable in large cluster > --- > > Key: YARN-6523 > URL: https://issues.apache.org/jira/browse/YARN-6523 > Project: Hadoop YARN > Issue Type: Improvement > Components: RM >Affects Versions: 2.8.0, 2.7.3 >Reporter: Naganarasimha G R >Assignee: Manikandan R >Priority: Major > Attachments: YARN-6523.001.patch, YARN-6523.002.patch, > YARN-6523.003.patch, YARN-6523.004.patch, YARN-6523.005.patch, > YARN-6523.006.patch, YARN-6523.007.patch, YARN-6523.008.patch, > YARN-6523.009.patch > > > Currently as part of heartbeat response RM sets all application's tokens > though all applications might not be active on the node. On top of it > NodeHeartbeatResponsePBImpl converts tokens for each app into > SystemCredentialsForAppsProto. Hence for each node and each heartbeat too > many SystemCredentialsForAppsProto objects were getting created. > We hit a OOM while testing for 2000 concurrent apps on 500 nodes cluster with > 8GB RAM configured for RM -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6523) Newly retrieved security Tokens are sent as part of each heartbeat to each node from RM which is not desirable in large cluster
[ https://issues.apache.org/jira/browse/YARN-6523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikandan R updated YARN-6523: --- Attachment: YARN-6523.009.patch > Newly retrieved security Tokens are sent as part of each heartbeat to each > node from RM which is not desirable in large cluster > --- > > Key: YARN-6523 > URL: https://issues.apache.org/jira/browse/YARN-6523 > Project: Hadoop YARN > Issue Type: Improvement > Components: RM >Affects Versions: 2.8.0, 2.7.3 >Reporter: Naganarasimha G R >Assignee: Manikandan R >Priority: Major > Attachments: YARN-6523.001.patch, YARN-6523.002.patch, > YARN-6523.003.patch, YARN-6523.004.patch, YARN-6523.005.patch, > YARN-6523.006.patch, YARN-6523.007.patch, YARN-6523.008.patch, > YARN-6523.009.patch > > > Currently as part of heartbeat response RM sets all application's tokens > though all applications might not be active on the node. On top of it > NodeHeartbeatResponsePBImpl converts tokens for each app into > SystemCredentialsForAppsProto. Hence for each node and each heartbeat too > many SystemCredentialsForAppsProto objects were getting created. > We hit a OOM while testing for 2000 concurrent apps on 500 nodes cluster with > 8GB RAM configured for RM -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6523) Newly retrieved security Tokens are sent as part of each heartbeat to each node from RM which is not desirable in large cluster
[
https://issues.apache.org/jira/browse/YARN-6523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16706479#comment-16706479
]
Hadoop QA commented on YARN-6523:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 14s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
27s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 7m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
44s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 32s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 14 new + 338 unchanged - 16 fixed = 352 total (was 354) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 13s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
45s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
45s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m
31s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}110m 28s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}229m 59s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-6523 |
| JIRA Patch URL |
[jira] [Commented] (YARN-9079) Speculative attempts should not run on the same node
[ https://issues.apache.org/jira/browse/YARN-9079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16706607#comment-16706607 ] zhaodong commented on YARN-9079: yeah, This issue is very important for us. > Speculative attempts should not run on the same node > > > Key: YARN-9079 > URL: https://issues.apache.org/jira/browse/YARN-9079 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, capacity scheduler, yarn >Affects Versions: 2.7.2 >Reporter: Lee chen >Priority: Major > Attachments: image-2018-12-03-09-54-07-859.png > > > I found in all versions of yarn, Speculative Execution may set the > speculative task to the node of original task.What i have read is only it > will try to have one more task attempt. haven't seen any place mentioning not > on same node.It is unreasonable.If the node have some problems lead to tasks > execution will be very slow. and then placement the speculative task to same > node cannot help the problematic task. > In our cluster （version 2.7.2，2700 nodes），this phenomenon appear > almost everyday. > !image-2018-12-03-09-54-07-859.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-8770) [Submarine] Support using Submarine to submit Pytorch job
[ https://issues.apache.org/jira/browse/YARN-8770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xun Liu reassigned YARN-8770: - Assignee: Xun Liu (was: Sunil Govindan) > [Submarine] Support using Submarine to submit Pytorch job > - > > Key: YARN-8770 > URL: https://issues.apache.org/jira/browse/YARN-8770 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Xun Liu >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9079) Speculative attempts should not run on the same node
Lee chen created YARN-9079: -- Summary: Speculative attempts should not run on the same node Key: YARN-9079 URL: https://issues.apache.org/jira/browse/YARN-9079 Project: Hadoop YARN Issue Type: New Feature Components: api, capacity scheduler, yarn Affects Versions: 2.7.2 Reporter: Lee chen Attachments: image-2018-12-03-09-54-07-859.png I found in all versions of yarn, Speculative Execution may set the speculative task to the node of original task.What i have read is only it will try to have one more task attempt. haven't seen any place mentioning not on same node.It is unreasonable.If the node have some problems lead to tasks execution will be very slow. and then placement the speculative task to same node cannot help the problematic task. In our cluster （version 2.7.2，2700 nodes），this phenomenon appear almost everyday. !image-2018-12-03-09-54-07-859.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8914) Add xtermjs to YARN UI2
[
https://issues.apache.org/jira/browse/YARN-8914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16706715#comment-16706715
]
Akhil PB edited comment on YARN-8914 at 12/3/18 6:58 AM:
-
[~eyang] I have found a few issues.
# The Terminal column in attempts table in {{timeline-view.js}} is not a link,
it's a plain text now. Please change this to a link.
# Please add the changes made for {{yarn-container.js}} model and serializer to
{{yarn-timeline-container.js}} model and serializer too.
# To access requestedUser, you could use the following code to add userInfo to
{{yarn-app-attempt.js}} model. Then we can pass in this model to
{{timeline-view.js}}.
{code}
// file: routes/yarn-app-attempt.js
// Adding userInfo to the model
afterModel(model) {
const appContrl = this.controllerFor('application');
model.userInfo = appContrl.get('userInfo');
}
{code}
{code}
// file: templates/yarn-app-attempt.js
// Passing userInfo to the timeline-view.js
{{timeline-view
parent-id="containers-timeline-div"
my-id="timeline-view"
height="400"
rmModel=model.rmContainers
tsModel=model.tsContainers
label="shortAppAttemptId"
attemptModel=false
changeViewType="changeViewType"
viewType=viewType
userInfo=model.userInfo
}}
{code}
Now we could access requested user via following:
{code}
let requestedUser = ''; // could assign some default user.
if (this.get('userInfo')) {
requestedUser = this.get('userInfo.requestedUser')
}
{code}
was (Author: akhilpb):
[~eyang] I have found a few issues.
# The Terminal column in attempts table in {{timeline-view.js}} is not a link,
it's a plain text now. Please change this to a link.
# Please add the changes made for {{yarn-container.js}} model and serializer to
{{yarn-timeline-container.js}} model and serializer too.
# For Container ID column in {{timeline-view.js}}, please use
{{row.get('requestedUser')}} to get the user. So that if we want to change the
implementation of accessing the user, we have to change only in model class.
> Add xtermjs to YARN UI2
> ---
>
> Key: YARN-8914
> URL: https://issues.apache.org/jira/browse/YARN-8914
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn-ui-v2
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-8914.001.patch, YARN-8914.002.patch,
> YARN-8914.003.patch, YARN-8914.004.patch, YARN-8914.005.patch,
> YARN-8914.006.patch, YARN-8914.007.patch
>
>
> In the container listing from UI2, we can add a link to connect to docker
> container using xtermjs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8914) Add xtermjs to YARN UI2
[
https://issues.apache.org/jira/browse/YARN-8914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16706715#comment-16706715
]
Akhil PB edited comment on YARN-8914 at 12/3/18 6:59 AM:
-
[~eyang] I have found a few issues.
# The Terminal column in attempts table in {{timeline-view.js}} is not a link,
it's a plain text now. Please change this to a link.
# Please add the changes made for {{yarn-container.js}} model and serializer to
{{yarn-timeline-container.js}} model and serializer too.
# To access requestedUser, you could use the following code to add userInfo to
{{yarn-app-attempt.js}} model. Then we can pass in this model to
{{timeline-view.js}}.
{code}
// file: routes/yarn-app-attempt.js
// Adding userInfo to the model
afterModel(model) {
const appContrl = this.controllerFor('application');
model.userInfo = appContrl.get('userInfo');
}
{code}
{code}
// file: templates/yarn-app-attempt.js
// Passing userInfo to the timeline-view.js
{{timeline-view
parent-id="containers-timeline-div"
my-id="timeline-view"
height="400"
rmModel=model.rmContainers
tsModel=model.tsContainers
label="shortAppAttemptId"
attemptModel=false
changeViewType="changeViewType"
viewType=viewType
userInfo=model.userInfo
}}
{code}
Now we could access requested user via following in {{timeline-view.js}}:
{code}
let requestedUser = ''; // could assign some default user.
if (this.get('userInfo')) {
requestedUser = this.get('userInfo.requestedUser')
}
{code}
was (Author: akhilpb):
[~eyang] I have found a few issues.
# The Terminal column in attempts table in {{timeline-view.js}} is not a link,
it's a plain text now. Please change this to a link.
# Please add the changes made for {{yarn-container.js}} model and serializer to
{{yarn-timeline-container.js}} model and serializer too.
# To access requestedUser, you could use the following code to add userInfo to
{{yarn-app-attempt.js}} model. Then we can pass in this model to
{{timeline-view.js}}.
{code}
// file: routes/yarn-app-attempt.js
// Adding userInfo to the model
afterModel(model) {
const appContrl = this.controllerFor('application');
model.userInfo = appContrl.get('userInfo');
}
{code}
{code}
// file: templates/yarn-app-attempt.js
// Passing userInfo to the timeline-view.js
{{timeline-view
parent-id="containers-timeline-div"
my-id="timeline-view"
height="400"
rmModel=model.rmContainers
tsModel=model.tsContainers
label="shortAppAttemptId"
attemptModel=false
changeViewType="changeViewType"
viewType=viewType
userInfo=model.userInfo
}}
{code}
Now we could access requested user via following:
{code}
let requestedUser = ''; // could assign some default user.
if (this.get('userInfo')) {
requestedUser = this.get('userInfo.requestedUser')
}
{code}
> Add xtermjs to YARN UI2
> ---
>
> Key: YARN-8914
> URL: https://issues.apache.org/jira/browse/YARN-8914
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn-ui-v2
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-8914.001.patch, YARN-8914.002.patch,
> YARN-8914.003.patch, YARN-8914.004.patch, YARN-8914.005.patch,
> YARN-8914.006.patch, YARN-8914.007.patch
>
>
> In the container listing from UI2, we can add a link to connect to docker
> container using xtermjs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8914) Add xtermjs to YARN UI2
[
https://issues.apache.org/jira/browse/YARN-8914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16706715#comment-16706715
]
Akhil PB edited comment on YARN-8914 at 12/3/18 7:00 AM:
-
[~eyang] I have found a few issues.
# The Terminal column in attempts table in {{timeline-view.js}} is not a link,
it's a plain text now. Please change this to a link.
# Please add the changes made for {{yarn-container.js}} model and serializer to
{{yarn-timeline-container.js}} model and serializer too.
# Please remove the requestedUser DOM access and from {{yarn-container.js}}
model.
# To access requestedUser, you could use the following code to add userInfo to
{{yarn-app-attempt.js}} model. Then we can pass in this model to
{{timeline-view.js}}.
{code}
// file: routes/yarn-app-attempt.js
// Adding userInfo to the model
afterModel(model) {
const appContrl = this.controllerFor('application');
model.userInfo = appContrl.get('userInfo');
}
{code}
{code}
// file: templates/yarn-app-attempt.js
// Passing userInfo to the timeline-view.js
{{timeline-view
parent-id="containers-timeline-div"
my-id="timeline-view"
height="400"
rmModel=model.rmContainers
tsModel=model.tsContainers
label="shortAppAttemptId"
attemptModel=false
changeViewType="changeViewType"
viewType=viewType
userInfo=model.userInfo
}}
{code}
Now we could access requested user via following in {{timeline-view.js}}:
{code}
let requestedUser = ''; // could assign some default user.
if (this.get('userInfo')) {
requestedUser = this.get('userInfo.requestedUser')
}
{code}
was (Author: akhilpb):
[~eyang] I have found a few issues.
# The Terminal column in attempts table in {{timeline-view.js}} is not a link,
it's a plain text now. Please change this to a link.
# Please add the changes made for {{yarn-container.js}} model and serializer to
{{yarn-timeline-container.js}} model and serializer too.
# To access requestedUser, you could use the following code to add userInfo to
{{yarn-app-attempt.js}} model. Then we can pass in this model to
{{timeline-view.js}}.
{code}
// file: routes/yarn-app-attempt.js
// Adding userInfo to the model
afterModel(model) {
const appContrl = this.controllerFor('application');
model.userInfo = appContrl.get('userInfo');
}
{code}
{code}
// file: templates/yarn-app-attempt.js
// Passing userInfo to the timeline-view.js
{{timeline-view
parent-id="containers-timeline-div"
my-id="timeline-view"
height="400"
rmModel=model.rmContainers
tsModel=model.tsContainers
label="shortAppAttemptId"
attemptModel=false
changeViewType="changeViewType"
viewType=viewType
userInfo=model.userInfo
}}
{code}
Now we could access requested user via following in {{timeline-view.js}}:
{code}
let requestedUser = ''; // could assign some default user.
if (this.get('userInfo')) {
requestedUser = this.get('userInfo.requestedUser')
}
{code}
> Add xtermjs to YARN UI2
> ---
>
> Key: YARN-8914
> URL: https://issues.apache.org/jira/browse/YARN-8914
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn-ui-v2
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-8914.001.patch, YARN-8914.002.patch,
> YARN-8914.003.patch, YARN-8914.004.patch, YARN-8914.005.patch,
> YARN-8914.006.patch, YARN-8914.007.patch
>
>
> In the container listing from UI2, we can add a link to connect to docker
> container using xtermjs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8937) TestLeaderElectorService hangs
[ https://issues.apache.org/jira/browse/YARN-8937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16706662#comment-16706662 ] Akira Ajisaka commented on YARN-8937: - I tried Curator 2.13.0 release candidate and the test succeeded. https://lists.apache.org/thread.html/959d9897e18ed628a46be06f2d4d2657b85998a2a083e0936f4bb0a9@%3Cdev.curator.apache.org%3E I'll provide a patch to bump the curator version after 2.13.0 is released. > TestLeaderElectorService hangs > -- > > Key: YARN-8937 > URL: https://issues.apache.org/jira/browse/YARN-8937 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.3.0 >Reporter: Jason Lowe >Priority: Major > > TestLeaderElectorService hangs waiting for the TestingZooKeeperServer to > start and eventually gets killed by the surefire timeout. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-8937) TestLeaderElectorService hangs
[ https://issues.apache.org/jira/browse/YARN-8937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka reassigned YARN-8937: --- Assignee: Akira Ajisaka > TestLeaderElectorService hangs > -- > > Key: YARN-8937 > URL: https://issues.apache.org/jira/browse/YARN-8937 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.3.0 >Reporter: Jason Lowe >Assignee: Akira Ajisaka >Priority: Major > > TestLeaderElectorService hangs waiting for the TestingZooKeeperServer to > start and eventually gets killed by the surefire timeout. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8914) Add xtermjs to YARN UI2
[
https://issues.apache.org/jira/browse/YARN-8914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16706715#comment-16706715
]
Akhil PB commented on YARN-8914:
[~eyang] I have found a couple of issues.
# The Terminal column in attempts table is not a link. Please change this to a
link.
# Please add the changes made for {{yarn-container.js}} model and serializer to
{{yarn-timeline-container.js}} model and serializer too.
> Add xtermjs to YARN UI2
> ---
>
> Key: YARN-8914
> URL: https://issues.apache.org/jira/browse/YARN-8914
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn-ui-v2
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-8914.001.patch, YARN-8914.002.patch,
> YARN-8914.003.patch, YARN-8914.004.patch, YARN-8914.005.patch,
> YARN-8914.006.patch, YARN-8914.007.patch
>
>
> In the container listing from UI2, we can add a link to connect to docker
> container using xtermjs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8914) Add xtermjs to YARN UI2
[
https://issues.apache.org/jira/browse/YARN-8914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16706715#comment-16706715
]
Akhil PB edited comment on YARN-8914 at 12/3/18 6:15 AM:
-
[~eyang] I have found a couple of issues.
# The Terminal column in attempts table is not a link, it's a plain text now.
Please change this to a link.
# Please add the changes made for {{yarn-container.js}} model and serializer to
{{yarn-timeline-container.js}} model and serializer too.
was (Author: akhilpb):
[~eyang] I have found a couple of issues.
# The Terminal column in attempts table is not a link. Please change this to a
link.
# Please add the changes made for {{yarn-container.js}} model and serializer to
{{yarn-timeline-container.js}} model and serializer too.
> Add xtermjs to YARN UI2
> ---
>
> Key: YARN-8914
> URL: https://issues.apache.org/jira/browse/YARN-8914
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn-ui-v2
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-8914.001.patch, YARN-8914.002.patch,
> YARN-8914.003.patch, YARN-8914.004.patch, YARN-8914.005.patch,
> YARN-8914.006.patch, YARN-8914.007.patch
>
>
> In the container listing from UI2, we can add a link to connect to docker
> container using xtermjs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8914) Add xtermjs to YARN UI2
[
https://issues.apache.org/jira/browse/YARN-8914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16706715#comment-16706715
]
Akhil PB edited comment on YARN-8914 at 12/3/18 6:26 AM:
-
[~eyang] I have found a few issues.
# The Terminal column in attempts table is not a link, it's a plain text now.
Please change this to a link.
# Please add the changes made for {{yarn-container.js}} model and serializer to
{{yarn-timeline-container.js}} model and serializer too.
# For Container ID column, please use {{row.get('requestedUser')}} to get the
user. So that if we want to change the implementation of accessing the user, we
have to change only in model class.
was (Author: akhilpb):
[~eyang] I have found a couple of issues.
# The Terminal column in attempts table is not a link, it's a plain text now.
Please change this to a link.
# Please add the changes made for {{yarn-container.js}} model and serializer to
{{yarn-timeline-container.js}} model and serializer too.
# For Container ID column, please use {{row.get('requestedUser')}} to get the
user. So that if we want to change the implementation of accessing the user, we
have to change only in model class.
> Add xtermjs to YARN UI2
> ---
>
> Key: YARN-8914
> URL: https://issues.apache.org/jira/browse/YARN-8914
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn-ui-v2
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-8914.001.patch, YARN-8914.002.patch,
> YARN-8914.003.patch, YARN-8914.004.patch, YARN-8914.005.patch,
> YARN-8914.006.patch, YARN-8914.007.patch
>
>
> In the container listing from UI2, we can add a link to connect to docker
> container using xtermjs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8914) Add xtermjs to YARN UI2
[
https://issues.apache.org/jira/browse/YARN-8914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16706715#comment-16706715
]
Akhil PB edited comment on YARN-8914 at 12/3/18 6:27 AM:
-
[~eyang] I have found a few issues.
# The Terminal column in attempts table is not a link, it's a plain text now.
Please change this to a link.
# Please add the changes made for {{yarn-container.js}} model and serializer to
{{yarn-timeline-container.js}} model and serializer too.
# For Container ID column in {{timeline-view.js}}, please use
{{row.get('requestedUser')}} to get the user. So that if we want to change the
implementation of accessing the user, we have to change only in model class.
was (Author: akhilpb):
[~eyang] I have found a few issues.
# The Terminal column in attempts table is not a link, it's a plain text now.
Please change this to a link.
# Please add the changes made for {{yarn-container.js}} model and serializer to
{{yarn-timeline-container.js}} model and serializer too.
# For Container ID column, please use {{row.get('requestedUser')}} to get the
user. So that if we want to change the implementation of accessing the user, we
have to change only in model class.
> Add xtermjs to YARN UI2
> ---
>
> Key: YARN-8914
> URL: https://issues.apache.org/jira/browse/YARN-8914
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn-ui-v2
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-8914.001.patch, YARN-8914.002.patch,
> YARN-8914.003.patch, YARN-8914.004.patch, YARN-8914.005.patch,
> YARN-8914.006.patch, YARN-8914.007.patch
>
>
> In the container listing from UI2, we can add a link to connect to docker
> container using xtermjs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8914) Add xtermjs to YARN UI2
[
https://issues.apache.org/jira/browse/YARN-8914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16706715#comment-16706715
]
Akhil PB edited comment on YARN-8914 at 12/3/18 6:26 AM:
-
[~eyang] I have found a couple of issues.
# The Terminal column in attempts table is not a link, it's a plain text now.
Please change this to a link.
# Please add the changes made for {{yarn-container.js}} model and serializer to
{{yarn-timeline-container.js}} model and serializer too.
# For Container ID column, please use {{row.get('requestedUser')}} to get the
user. So that if we want to change the implementation of accessing the user, we
have to change only in model class.
was (Author: akhilpb):
[~eyang] I have found a couple of issues.
# The Terminal column in attempts table is not a link, it's a plain text now.
Please change this to a link.
# Please add the changes made for {{yarn-container.js}} model and serializer to
{{yarn-timeline-container.js}} model and serializer too.
> Add xtermjs to YARN UI2
> ---
>
> Key: YARN-8914
> URL: https://issues.apache.org/jira/browse/YARN-8914
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn-ui-v2
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-8914.001.patch, YARN-8914.002.patch,
> YARN-8914.003.patch, YARN-8914.004.patch, YARN-8914.005.patch,
> YARN-8914.006.patch, YARN-8914.007.patch
>
>
> In the container listing from UI2, we can add a link to connect to docker
> container using xtermjs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8914) Add xtermjs to YARN UI2
[
https://issues.apache.org/jira/browse/YARN-8914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16706715#comment-16706715
]
Akhil PB edited comment on YARN-8914 at 12/3/18 6:27 AM:
-
[~eyang] I have found a few issues.
# The Terminal column in attempts table in {{timeline-view.js}} is not a link,
it's a plain text now. Please change this to a link.
# Please add the changes made for {{yarn-container.js}} model and serializer to
{{yarn-timeline-container.js}} model and serializer too.
# For Container ID column in {{timeline-view.js}}, please use
{{row.get('requestedUser')}} to get the user. So that if we want to change the
implementation of accessing the user, we have to change only in model class.
was (Author: akhilpb):
[~eyang] I have found a few issues.
# The Terminal column in attempts table is not a link, it's a plain text now.
Please change this to a link.
# Please add the changes made for {{yarn-container.js}} model and serializer to
{{yarn-timeline-container.js}} model and serializer too.
# For Container ID column in {{timeline-view.js}}, please use
{{row.get('requestedUser')}} to get the user. So that if we want to change the
implementation of accessing the user, we have to change only in model class.
> Add xtermjs to YARN UI2
> ---
>
> Key: YARN-8914
> URL: https://issues.apache.org/jira/browse/YARN-8914
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn-ui-v2
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-8914.001.patch, YARN-8914.002.patch,
> YARN-8914.003.patch, YARN-8914.004.patch, YARN-8914.005.patch,
> YARN-8914.006.patch, YARN-8914.007.patch
>
>
> In the container listing from UI2, we can add a link to connect to docker
> container using xtermjs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8914) Add xtermjs to YARN UI2
[
https://issues.apache.org/jira/browse/YARN-8914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16706715#comment-16706715
]
Akhil PB edited comment on YARN-8914 at 12/3/18 7:01 AM:
-
[~eyang] I have found a few issues.
# The Terminal column in attempts table in {{timeline-view.js}} is not a link,
it's a plain text now. Please change this to a link. I suppose this is intended
to access AM container.
# Please add the changes made for {{yarn-container.js}} model and serializer to
{{yarn-timeline-container.js}} model and serializer too.
# Please remove the requestedUser DOM access and from {{yarn-container.js}}
model.
# To access requestedUser, you could use the following code to add userInfo to
{{yarn-app-attempt.js}} model. Then we can pass in this model to
{{timeline-view.js}}.
{code}
// file: routes/yarn-app-attempt.js
// Adding userInfo to the model
afterModel(model) {
const appContrl = this.controllerFor('application');
model.userInfo = appContrl.get('userInfo');
}
{code}
{code}
// file: templates/yarn-app-attempt.js
// Passing userInfo to the timeline-view.js
{{timeline-view
parent-id="containers-timeline-div"
my-id="timeline-view"
height="400"
rmModel=model.rmContainers
tsModel=model.tsContainers
label="shortAppAttemptId"
attemptModel=false
changeViewType="changeViewType"
viewType=viewType
userInfo=model.userInfo
}}
{code}
Now we could access requested user via following in {{timeline-view.js}}:
{code}
let requestedUser = ''; // could assign some default user.
if (this.get('userInfo')) {
requestedUser = this.get('userInfo.requestedUser')
}
{code}
was (Author: akhilpb):
[~eyang] I have found a few issues.
# The Terminal column in attempts table in {{timeline-view.js}} is not a link,
it's a plain text now. Please change this to a link.
# Please add the changes made for {{yarn-container.js}} model and serializer to
{{yarn-timeline-container.js}} model and serializer too.
# Please remove the requestedUser DOM access and from {{yarn-container.js}}
model.
# To access requestedUser, you could use the following code to add userInfo to
{{yarn-app-attempt.js}} model. Then we can pass in this model to
{{timeline-view.js}}.
{code}
// file: routes/yarn-app-attempt.js
// Adding userInfo to the model
afterModel(model) {
const appContrl = this.controllerFor('application');
model.userInfo = appContrl.get('userInfo');
}
{code}
{code}
// file: templates/yarn-app-attempt.js
// Passing userInfo to the timeline-view.js
{{timeline-view
parent-id="containers-timeline-div"
my-id="timeline-view"
height="400"
rmModel=model.rmContainers
tsModel=model.tsContainers
label="shortAppAttemptId"
attemptModel=false
changeViewType="changeViewType"
viewType=viewType
userInfo=model.userInfo
}}
{code}
Now we could access requested user via following in {{timeline-view.js}}:
{code}
let requestedUser = ''; // could assign some default user.
if (this.get('userInfo')) {
requestedUser = this.get('userInfo.requestedUser')
}
{code}
> Add xtermjs to YARN UI2
> ---
>
> Key: YARN-8914
> URL: https://issues.apache.org/jira/browse/YARN-8914
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn-ui-v2
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-8914.001.patch, YARN-8914.002.patch,
> YARN-8914.003.patch, YARN-8914.004.patch, YARN-8914.005.patch,
> YARN-8914.006.patch, YARN-8914.007.patch
>
>
> In the container listing from UI2, we can add a link to connect to docker
> container using xtermjs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8914) Add xtermjs to YARN UI2
[
https://issues.apache.org/jira/browse/YARN-8914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16706715#comment-16706715
]
Akhil PB edited comment on YARN-8914 at 12/3/18 7:14 AM:
-
[~eyang] I have found a few issues.
# The Terminal column in attempts table in {{timeline-view.js}} is not a link,
it's a plain text now. Please change this to a link. I suppose this is intended
to access AM container.
# Please add the changes made for {{yarn-container.js}} model and serializer to
{{yarn-timeline-container.js}} model and serializer too.
# Please remove the requestedUser DOM access and from {{yarn-container.js}}
model.
# To access requestedUser, you could use the following code to add userInfo to
{{yarn-app-attempt.js}} model. Then we can pass in this model to
{{timeline-view.js}}.
{code}
// file: routes/yarn-app-attempt.js
// Adding userInfo to the model
afterModel(model) {
const appContrl = this.controllerFor('application');
model.userInfo = appContrl.get('userInfo');
}
{code}
{code}
// file: templates/yarn-app-attempt.js
// Passing userInfo to the timeline-view.js
{{timeline-view
parent-id="containers-timeline-div"
my-id="timeline-view"
height="400"
rmModel=model.rmContainers
tsModel=model.tsContainers
label="shortAppAttemptId"
attemptModel=false
changeViewType="changeViewType"
viewType=viewType
userInfo=model.userInfo
}}
{code}
Now we could access requested user by {{this.get('requestedUser')}} in
{{timeline-view.js}} as following:
{code}
requestedUser: function() {
if (this.get('userInfo')) {
return this.get('userInfo.requestedUser');
}
return ''; // could return some default uer
}.property('userInfo'),
{code}
Now you can pass in userInfo to {{templates/components/container-table.hbs}}:
{code}
{{container-table container=selected requestedUser=requestedUser}}
{code}
was (Author: akhilpb):
[~eyang] I have found a few issues.
# The Terminal column in attempts table in {{timeline-view.js}} is not a link,
it's a plain text now. Please change this to a link. I suppose this is intended
to access AM container.
# Please add the changes made for {{yarn-container.js}} model and serializer to
{{yarn-timeline-container.js}} model and serializer too.
# Please remove the requestedUser DOM access and from {{yarn-container.js}}
model.
# To access requestedUser, you could use the following code to add userInfo to
{{yarn-app-attempt.js}} model. Then we can pass in this model to
{{timeline-view.js}}.
{code}
// file: routes/yarn-app-attempt.js
// Adding userInfo to the model
afterModel(model) {
const appContrl = this.controllerFor('application');
model.userInfo = appContrl.get('userInfo');
}
{code}
{code}
// file: templates/yarn-app-attempt.js
// Passing userInfo to the timeline-view.js
{{timeline-view
parent-id="containers-timeline-div"
my-id="timeline-view"
height="400"
rmModel=model.rmContainers
tsModel=model.tsContainers
label="shortAppAttemptId"
attemptModel=false
changeViewType="changeViewType"
viewType=viewType
userInfo=model.userInfo
}}
{code}
Now we could access requested user via following in {{timeline-view.js}}:
{code}
let requestedUser = ''; // could assign some default user.
if (this.get('userInfo')) {
requestedUser = this.get('userInfo.requestedUser')
}
{code}
> Add xtermjs to YARN UI2
> ---
>
> Key: YARN-8914
> URL: https://issues.apache.org/jira/browse/YARN-8914
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn-ui-v2
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-8914.001.patch, YARN-8914.002.patch,
> YARN-8914.003.patch, YARN-8914.004.patch, YARN-8914.005.patch,
> YARN-8914.006.patch, YARN-8914.007.patch
>
>
> In the container listing from UI2, we can add a link to connect to docker
> container using xtermjs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8789) Add BoundedQueue to AsyncDispatcher
[
https://issues.apache.org/jira/browse/YARN-8789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16706529#comment-16706529
]
Wilfred Spiegelenburg commented on YARN-8789:
-
Hi [~belugabehr] can you explain a bit more about when you saw the issue and
which release you were on. [~pbacsko] already implemented a MR specific change
in MAPREDUCE-5124 which has helped our deployments a lot. In that change we
merge (coalesce) the events in the AM. It helps with the event queue growth as
not every event is a new entry. Do we need still need a bounded queue for the
events after that?
The other thing that I do not understand is how your fix would really help the
AM. It still needs to process all the events. You just stop adding them when
the queue is full. They do not disappear. I think that if it can not keep up
processing just stop accepting events does not solve the problem you just push
it further away.
> Add BoundedQueue to AsyncDispatcher
> ---
>
> Key: YARN-8789
> URL: https://issues.apache.org/jira/browse/YARN-8789
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: applications
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: YARN-8789.1.patch, YARN-8789.10.patch,
> YARN-8789.12.patch, YARN-8789.14.patch, YARN-8789.2.patch, YARN-8789.3.patch,
> YARN-8789.4.patch, YARN-8789.5.patch, YARN-8789.6.patch, YARN-8789.7.patch,
> YARN-8789.7.patch, YARN-8789.8.patch, YARN-8789.9.patch
>
>
> I recently came across a scenario where an MR ApplicationMaster was failing
> with an OOM exception. It had many thousands of Mappers and thousands of
> Reducers. It was noted that in the logging that the event-queue of
> {{AsyncDispatcher}} had a very large number of item in it and was seemingly
> never decreasing.
> I started looking at the code and thought it could use some clean up,
> simplification, and the ability to specify a bounded queue so that any
> incoming events are throttled until they can be processed. This will protect
> the ApplicationMaster from a flood of events.
> Logging Message:
> Size of event-queue is xxx
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
