[jira] [Commented] (YARN-9573) DistributedShell cannot specify LogAggregationContext
[ https://issues.apache.org/jira/browse/YARN-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16885311#comment-16885311 ] Adam Antal commented on YARN-9573: -- Thanks [~snemeth]! > DistributedShell cannot specify LogAggregationContext > - > > Key: YARN-9573 > URL: https://issues.apache.org/jira/browse/YARN-9573 > Project: Hadoop YARN > Issue Type: Improvement > Components: distributed-shell, log-aggregation, yarn >Affects Versions: 3.2.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Fix For: 3.3.0, 3.2.1, 3.1.3 > > Attachments: YARN-9573.001.branch-3.1.patch, > YARN-9573.001.branch-3.2.patch, YARN-9573.001.patch, YARN-9573.002.patch, > YARN-9573.002.patch, YARN-9573.003.patch > > > When DShell sends the application request object to the RM, it doesn't > specify the LogAggregationContext object - thus it is not possible to run > DShell with various log-aggregation configurations, for e.g. a rolling > fashioned log aggregation. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9573) DistributedShell cannot specify LogAggregationContext
[ https://issues.apache.org/jira/browse/YARN-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883224#comment-16883224 ] Szilard Nemeth commented on YARN-9573: -- Hi [~adam.antal]! Thanks for the patches, just pushed 3.2 / 3.1 patches. Thanks [~sunilg] for the reviews! > DistributedShell cannot specify LogAggregationContext > - > > Key: YARN-9573 > URL: https://issues.apache.org/jira/browse/YARN-9573 > Project: Hadoop YARN > Issue Type: Improvement > Components: distributed-shell, log-aggregation, yarn >Affects Versions: 3.2.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: YARN-9573.001.branch-3.1.patch, > YARN-9573.001.branch-3.2.patch, YARN-9573.001.patch, YARN-9573.002.patch, > YARN-9573.002.patch, YARN-9573.003.patch > > > When DShell sends the application request object to the RM, it doesn't > specify the LogAggregationContext object - thus it is not possible to run > DShell with various log-aggregation configurations, for e.g. a rolling > fashioned log aggregation. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9573) DistributedShell cannot specify LogAggregationContext
[ https://issues.apache.org/jira/browse/YARN-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866574#comment-16866574 ] Adam Antal commented on YARN-9573: -- [~sunilg], did you have a chance to check the patches for the other branches? > DistributedShell cannot specify LogAggregationContext > - > > Key: YARN-9573 > URL: https://issues.apache.org/jira/browse/YARN-9573 > Project: Hadoop YARN > Issue Type: Improvement > Components: distributed-shell, log-aggregation, yarn >Affects Versions: 3.2.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: YARN-9573.001.branch-3.1.patch, > YARN-9573.001.branch-3.2.patch, YARN-9573.001.patch, YARN-9573.002.patch, > YARN-9573.002.patch, YARN-9573.003.patch > > > When DShell sends the application request object to the RM, it doesn't > specify the LogAggregationContext object - thus it is not possible to run > DShell with various log-aggregation configurations, for e.g. a rolling > fashioned log aggregation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9573) DistributedShell cannot specify LogAggregationContext
[
https://issues.apache.org/jira/browse/YARN-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16857582#comment-16857582
]
Hadoop QA commented on YARN-9573:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-3.1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
38s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 59s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} branch-3.1 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 15s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell:
The patch generated 2 new + 157 unchanged - 3 fixed = 159 total (was 160)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 12s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 26s{color}
| {color:red} hadoop-yarn-applications-distributedshell in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
29s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 80m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.applications.distributedshell.TestDSWithMultipleNodeManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:080e9d0 |
| JIRA Issue | YARN-9573 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12971047/YARN-9573.001.branch-3.1.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux d541aa73e626 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-3.1 / d65371c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/24245/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt
|
| unit |
[jira] [Commented] (YARN-9573) DistributedShell cannot specify LogAggregationContext
[ https://issues.apache.org/jira/browse/YARN-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16857512#comment-16857512 ] Adam Antal commented on YARN-9573: -- Yes, I uploaded patches to branch-3.2: [^YARN-9573.001.branch-3.2.patch] and branch-3.1: [^YARN-9573.001.branch-3.1.patch]. Conflicts were due to the usage of VisibleForTesting annotation and the assertFalse both introduced by YARN-9227. > DistributedShell cannot specify LogAggregationContext > - > > Key: YARN-9573 > URL: https://issues.apache.org/jira/browse/YARN-9573 > Project: Hadoop YARN > Issue Type: Improvement > Components: distributed-shell, log-aggregation, yarn >Affects Versions: 3.2.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: YARN-9573.001.branch-3.2.patch, YARN-9573.001.patch, > YARN-9573.002.patch, YARN-9573.002.patch, YARN-9573.003.patch > > > When DShell sends the application request object to the RM, it doesn't > specify the LogAggregationContext object - thus it is not possible to run > DShell with various log-aggregation configurations, for e.g. a rolling > fashioned log aggregation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9573) DistributedShell cannot specify LogAggregationContext
[ https://issues.apache.org/jira/browse/YARN-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16857476#comment-16857476 ] Hudson commented on YARN-9573: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16686 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16686/]) YARN-9573. DistributedShell cannot specify LogAggregationContext. (sunilg: rev 649666e118a7cf92b676eaa56a8be318176c443e) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java > DistributedShell cannot specify LogAggregationContext > - > > Key: YARN-9573 > URL: https://issues.apache.org/jira/browse/YARN-9573 > Project: Hadoop YARN > Issue Type: Improvement > Components: distributed-shell, log-aggregation, yarn >Affects Versions: 3.2.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: YARN-9573.001.patch, YARN-9573.002.patch, > YARN-9573.002.patch, YARN-9573.003.patch > > > When DShell sends the application request object to the RM, it doesn't > specify the LogAggregationContext object - thus it is not possible to run > DShell with various log-aggregation configurations, for e.g. a rolling > fashioned log aggregation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9573) DistributedShell cannot specify LogAggregationContext
[ https://issues.apache.org/jira/browse/YARN-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16857468#comment-16857468 ] Sunil Govindan commented on YARN-9573: -- back ported to trunk cud u pls help to backport to branch-3.2/3.1 > DistributedShell cannot specify LogAggregationContext > - > > Key: YARN-9573 > URL: https://issues.apache.org/jira/browse/YARN-9573 > Project: Hadoop YARN > Issue Type: Improvement > Components: distributed-shell, log-aggregation, yarn >Affects Versions: 3.2.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: YARN-9573.001.patch, YARN-9573.002.patch, > YARN-9573.002.patch, YARN-9573.003.patch > > > When DShell sends the application request object to the RM, it doesn't > specify the LogAggregationContext object - thus it is not possible to run > DShell with various log-aggregation configurations, for e.g. a rolling > fashioned log aggregation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9573) DistributedShell cannot specify LogAggregationContext
[ https://issues.apache.org/jira/browse/YARN-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16857460#comment-16857460 ] Adam Antal commented on YARN-9573: -- [~sunilg] all items have been addressed. Could you take a look at it? Thanks! > DistributedShell cannot specify LogAggregationContext > - > > Key: YARN-9573 > URL: https://issues.apache.org/jira/browse/YARN-9573 > Project: Hadoop YARN > Issue Type: Improvement > Components: distributed-shell, log-aggregation, yarn >Affects Versions: 3.2.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: YARN-9573.001.patch, YARN-9573.002.patch, > YARN-9573.002.patch, YARN-9573.003.patch > > > When DShell sends the application request object to the RM, it doesn't > specify the LogAggregationContext object - thus it is not possible to run > DShell with various log-aggregation configurations, for e.g. a rolling > fashioned log aggregation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9573) DistributedShell cannot specify LogAggregationContext
[ https://issues.apache.org/jira/browse/YARN-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16856580#comment-16856580 ] Szilard Nemeth commented on YARN-9573: -- Hi [~adam.antal]! Latest patch +1 (non-binding)! > DistributedShell cannot specify LogAggregationContext > - > > Key: YARN-9573 > URL: https://issues.apache.org/jira/browse/YARN-9573 > Project: Hadoop YARN > Issue Type: Improvement > Components: distributed-shell, log-aggregation, yarn >Affects Versions: 3.2.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: YARN-9573.001.patch, YARN-9573.002.patch, > YARN-9573.002.patch, YARN-9573.003.patch > > > When DShell sends the application request object to the RM, it doesn't > specify the LogAggregationContext object - thus it is not possible to run > DShell with various log-aggregation configurations, for e.g. a rolling > fashioned log aggregation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9573) DistributedShell cannot specify LogAggregationContext
[ https://issues.apache.org/jira/browse/YARN-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16855847#comment-16855847 ] Adam Antal commented on YARN-9573: -- [~sunilg] that is the other checkstyle error: the too long line. I didn't touch that line so I would not break it into two lines, if you don't insist. > DistributedShell cannot specify LogAggregationContext > - > > Key: YARN-9573 > URL: https://issues.apache.org/jira/browse/YARN-9573 > Project: Hadoop YARN > Issue Type: Improvement > Components: distributed-shell, log-aggregation, yarn >Affects Versions: 3.2.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: YARN-9573.001.patch, YARN-9573.002.patch, > YARN-9573.002.patch, YARN-9573.003.patch > > > When DShell sends the application request object to the RM, it doesn't > specify the LogAggregationContext object - thus it is not possible to run > DShell with various log-aggregation configurations, for e.g. a rolling > fashioned log aggregation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9573) DistributedShell cannot specify LogAggregationContext
[
https://issues.apache.org/jira/browse/YARN-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16855811#comment-16855811
]
Hadoop QA commented on YARN-9573:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 56s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 11s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell:
The patch generated 1 new + 157 unchanged - 3 fixed = 158 total (was 160)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 35s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 20m
33s{color} | {color:green} hadoop-yarn-applications-distributedshell in the
patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 66m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9573 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12970840/YARN-9573.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 43a2fcd2359f 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 7991159 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/24226/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24226/testReport/ |
| Max. process+thread count | 653 (vs. ulimit of 1) |
| modules | C:
[jira] [Commented] (YARN-9573) DistributedShell cannot specify LogAggregationContext
[ https://issues.apache.org/jira/browse/YARN-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16855746#comment-16855746 ] Sunil Govindan commented on YARN-9573: -- Thanks [~adam.antal] I definitely would love to see removal of unused imports. This makes code more cleaner. > DistributedShell cannot specify LogAggregationContext > - > > Key: YARN-9573 > URL: https://issues.apache.org/jira/browse/YARN-9573 > Project: Hadoop YARN > Issue Type: Improvement > Components: distributed-shell, log-aggregation, yarn >Affects Versions: 3.2.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: YARN-9573.001.patch, YARN-9573.002.patch, > YARN-9573.002.patch, YARN-9573.003.patch > > > When DShell sends the application request object to the RM, it doesn't > specify the LogAggregationContext object - thus it is not possible to run > DShell with various log-aggregation configurations, for e.g. a rolling > fashioned log aggregation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9573) DistributedShell cannot specify LogAggregationContext
[ https://issues.apache.org/jira/browse/YARN-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16855727#comment-16855727 ] Adam Antal commented on YARN-9573: -- One of the checkstyle is unrelated (didn't touch that line). If you insist, I'll fix that, but I'd pass. [~sunilg] could you please review this? > DistributedShell cannot specify LogAggregationContext > - > > Key: YARN-9573 > URL: https://issues.apache.org/jira/browse/YARN-9573 > Project: Hadoop YARN > Issue Type: Improvement > Components: distributed-shell, log-aggregation, yarn >Affects Versions: 3.2.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: YARN-9573.001.patch, YARN-9573.002.patch, > YARN-9573.002.patch, YARN-9573.003.patch > > > When DShell sends the application request object to the RM, it doesn't > specify the LogAggregationContext object - thus it is not possible to run > DShell with various log-aggregation configurations, for e.g. a rolling > fashioned log aggregation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9573) DistributedShell cannot specify LogAggregationContext
[
https://issues.apache.org/jira/browse/YARN-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16855702#comment-16855702
]
Hadoop QA commented on YARN-9573:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 35s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 12s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell:
The patch generated 2 new + 157 unchanged - 3 fixed = 159 total (was 160)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 56s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m
2s{color} | {color:green} hadoop-yarn-applications-distributedshell in the
patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9573 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12970826/YARN-9573.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux e44e6ed58428 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 7991159 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/24224/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24224/testReport/ |
| Max. process+thread count | 658 (vs. ulimit of 1) |
| modules | C:
[jira] [Commented] (YARN-9573) DistributedShell cannot specify LogAggregationContext
[ https://issues.apache.org/jira/browse/YARN-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16855640#comment-16855640 ] Adam Antal commented on YARN-9573: -- Jenkins still not triggered - uploaded patch v2 again, hoping it will trigger now. > DistributedShell cannot specify LogAggregationContext > - > > Key: YARN-9573 > URL: https://issues.apache.org/jira/browse/YARN-9573 > Project: Hadoop YARN > Issue Type: Improvement > Components: distributed-shell, log-aggregation, yarn >Affects Versions: 3.2.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: YARN-9573.001.patch, YARN-9573.002.patch, > YARN-9573.002.patch > > > When DShell sends the application request object to the RM, it doesn't > specify the LogAggregationContext object - thus it is not possible to run > DShell with various log-aggregation configurations, for e.g. a rolling > fashioned log aggregation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9573) DistributedShell cannot specify LogAggregationContext
[ https://issues.apache.org/jira/browse/YARN-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16855416#comment-16855416 ] Adam Antal commented on YARN-9573: -- Strangely jenkins didn't run. Retriggering it by moving this issue into Open, then Patch available status again. > DistributedShell cannot specify LogAggregationContext > - > > Key: YARN-9573 > URL: https://issues.apache.org/jira/browse/YARN-9573 > Project: Hadoop YARN > Issue Type: Improvement > Components: distributed-shell, log-aggregation, yarn >Affects Versions: 3.2.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: YARN-9573.001.patch, YARN-9573.002.patch > > > When DShell sends the application request object to the RM, it doesn't > specify the LogAggregationContext object - thus it is not possible to run > DShell with various log-aggregation configurations, for e.g. a rolling > fashioned log aggregation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9573) DistributedShell cannot specify LogAggregationContext
[ https://issues.apache.org/jira/browse/YARN-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16854648#comment-16854648 ] Prabhu Joseph commented on YARN-9573: - No [~adam.antal], will analyze this as part of YARN-9587. > DistributedShell cannot specify LogAggregationContext > - > > Key: YARN-9573 > URL: https://issues.apache.org/jira/browse/YARN-9573 > Project: Hadoop YARN > Issue Type: Improvement > Components: distributed-shell, log-aggregation, yarn >Affects Versions: 3.2.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: YARN-9573.001.patch, YARN-9573.002.patch > > > When DShell sends the application request object to the RM, it doesn't > specify the LogAggregationContext object - thus it is not possible to run > DShell with various log-aggregation configurations, for e.g. a rolling > fashioned log aggregation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9573) DistributedShell cannot specify LogAggregationContext
[ https://issues.apache.org/jira/browse/YARN-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16854624#comment-16854624 ] Adam Antal commented on YARN-9573: -- Added patchset v2 with removed \t characters and with a UT added as [~snemeth] suggested. I'd recommend fixing this because it has taken for me a LOT of overhead to search in that 800 MB of log file only because some stack trace is thrown in every 2-3 millisec. Do you have an open issue for this, [~Prabhu Joseph]? > DistributedShell cannot specify LogAggregationContext > - > > Key: YARN-9573 > URL: https://issues.apache.org/jira/browse/YARN-9573 > Project: Hadoop YARN > Issue Type: Improvement > Components: distributed-shell, log-aggregation, yarn >Affects Versions: 3.2.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: YARN-9573.001.patch, YARN-9573.002.patch > > > When DShell sends the application request object to the RM, it doesn't > specify the LogAggregationContext object - thus it is not possible to run > DShell with various log-aggregation configurations, for e.g. a rolling > fashioned log aggregation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9573) DistributedShell cannot specify LogAggregationContext
[ https://issues.apache.org/jira/browse/YARN-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16853327#comment-16853327 ] Prabhu Joseph commented on YARN-9573: - [~adam.antal] This happens in TestDistributedShell when testcase (Client) runs the job and waits for the result. This is ignorable. There will be separate AppMaster.stdout and stderr files which may have some clue. > DistributedShell cannot specify LogAggregationContext > - > > Key: YARN-9573 > URL: https://issues.apache.org/jira/browse/YARN-9573 > Project: Hadoop YARN > Issue Type: Improvement > Components: distributed-shell, log-aggregation, yarn >Affects Versions: 3.2.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: YARN-9573.001.patch > > > When DShell sends the application request object to the RM, it doesn't > specify the LogAggregationContext object - thus it is not possible to run > DShell with various log-aggregation configurations, for e.g. a rolling > fashioned log aggregation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9573) DistributedShell cannot specify LogAggregationContext
[ https://issues.apache.org/jira/browse/YARN-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16853175#comment-16853175 ] Szilard Nemeth commented on YARN-9573: -- Hi [~adam.antal]! Never seen this one before. [~wilfreds]: Have you seen such similar stacktrace before? > DistributedShell cannot specify LogAggregationContext > - > > Key: YARN-9573 > URL: https://issues.apache.org/jira/browse/YARN-9573 > Project: Hadoop YARN > Issue Type: Improvement > Components: distributed-shell, log-aggregation, yarn >Affects Versions: 3.2.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: YARN-9573.001.patch > > > When DShell sends the application request object to the RM, it doesn't > specify the LogAggregationContext object - thus it is not possible to run > DShell with various log-aggregation configurations, for e.g. a rolling > fashioned log aggregation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9573) DistributedShell cannot specify LogAggregationContext
[
https://issues.apache.org/jira/browse/YARN-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16853168#comment-16853168
]
Adam Antal commented on YARN-9573:
--
I couldn't get any further with this patch, because I couldn't start the
TestDistributedShell tests locally. The logs are always filled up with the
following stacktraces: (outputting like 800MB log during the 90sec timeout
period):
{noformat}
2019-05-31 17:54:08,972 WARN [IPC Server Responder] ipc.Server
(Server.java:doRunLoop(1546)) - Exception in Responder
java.io.IOException: Invalid argument
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:1485)
at org.apache.hadoop.ipc.Server$Responder.run(Server.java:1468)
{noformat}
This is probably some local issue at my computer. Anyone has seen something
similar to this before?
> DistributedShell cannot specify LogAggregationContext
> -
>
> Key: YARN-9573
> URL: https://issues.apache.org/jira/browse/YARN-9573
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: distributed-shell, log-aggregation, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: YARN-9573.001.patch
>
>
> When DShell sends the application request object to the RM, it doesn't
> specify the LogAggregationContext object - thus it is not possible to run
> DShell with various log-aggregation configurations, for e.g. a rolling
> fashioned log aggregation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9573) DistributedShell cannot specify LogAggregationContext
[ https://issues.apache.org/jira/browse/YARN-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852048#comment-16852048 ] Szilard Nemeth commented on YARN-9573: -- Hi [~adam.antal]! As discussed with [~sunilg], could you please add some unit tests for this change? Thanks! > DistributedShell cannot specify LogAggregationContext > - > > Key: YARN-9573 > URL: https://issues.apache.org/jira/browse/YARN-9573 > Project: Hadoop YARN > Issue Type: Improvement > Components: distributed-shell, log-aggregation, yarn >Affects Versions: 3.2.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: YARN-9573.001.patch > > > When DShell sends the application request object to the RM, it doesn't > specify the LogAggregationContext object - thus it is not possible to run > DShell with various log-aggregation configurations, for e.g. a rolling > fashioned log aggregation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9573) DistributedShell cannot specify LogAggregationContext
[ https://issues.apache.org/jira/browse/YARN-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847644#comment-16847644 ] Adam Antal commented on YARN-9573: -- Thanks for the reviews, just to make sure let's wait until YARN-9425 got comitted, don't want to accidentally mess up something in the background. > DistributedShell cannot specify LogAggregationContext > - > > Key: YARN-9573 > URL: https://issues.apache.org/jira/browse/YARN-9573 > Project: Hadoop YARN > Issue Type: Improvement > Components: distributed-shell, log-aggregation, yarn >Affects Versions: 3.2.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: YARN-9573.001.patch > > > When DShell sends the application request object to the RM, it doesn't > specify the LogAggregationContext object - thus it is not possible to run > DShell with various log-aggregation configurations, for e.g. a rolling > fashioned log aggregation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
