[jira] [Resolved] (YARN-9447) RM Crashes with NPE at TimelineServiceV2Publisher.putEntity
[
https://issues.apache.org/jira/browse/YARN-9447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph resolved YARN-9447.
-
Resolution: Duplicate
> RM Crashes with NPE at TimelineServiceV2Publisher.putEntity
> ---
>
> Key: YARN-9447
> URL: https://issues.apache.org/jira/browse/YARN-9447
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Critical
> Attachments: YARN-9447-001.patch, rm.log
>
>
> ResourceManager crashes with NullPointerException when
> TimelineServiceV2Publisher does putEntity after the timeline collector
> service for application is removed. This happened when killing a mapreduce
> job.
> {code}
> 2019-04-05 14:53:24,728 INFO
> org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollectorManager:
> The collector service for application_1553788280931_0013 was removed
> 2019-04-05 14:53:24,734 FATAL org.apache.hadoop.yarn.event.AsyncDispatcher:
> Error in dispatcher thread
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher.putEntity(TimelineServiceV2Publisher.java:461)
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher.access$100(TimelineServiceV2Publisher.java:73)
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher$TimelineV2EventHandler.handle(TimelineServiceV2Publisher.java:496)
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher$TimelineV2EventHandler.handle(TimelineServiceV2Publisher.java:485)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:748)
> 2019-04-05 14:53:24,743 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Exiting, bbye..
> 2019-04-05 14:53:24,758 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler:
> Container container_e30_1553788280931_0013_01_01 completed with event
> FINISHED, but corresponding RMContainer doesn't exist.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-9447) RM Crashes with NPE at TimelineServiceV2Publisher.putEntity
[
https://issues.apache.org/jira/browse/YARN-9447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph reopened YARN-9447:
-
> RM Crashes with NPE at TimelineServiceV2Publisher.putEntity
> ---
>
> Key: YARN-9447
> URL: https://issues.apache.org/jira/browse/YARN-9447
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Critical
> Attachments: YARN-9447-001.patch, rm.log
>
>
> ResourceManager crashes with NullPointerException when
> TimelineServiceV2Publisher does putEntity after the timeline collector
> service for application is removed. This happened when killing a mapreduce
> job.
> {code}
> 2019-04-05 14:53:24,728 INFO
> org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollectorManager:
> The collector service for application_1553788280931_0013 was removed
> 2019-04-05 14:53:24,734 FATAL org.apache.hadoop.yarn.event.AsyncDispatcher:
> Error in dispatcher thread
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher.putEntity(TimelineServiceV2Publisher.java:461)
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher.access$100(TimelineServiceV2Publisher.java:73)
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher$TimelineV2EventHandler.handle(TimelineServiceV2Publisher.java:496)
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher$TimelineV2EventHandler.handle(TimelineServiceV2Publisher.java:485)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:748)
> 2019-04-05 14:53:24,743 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Exiting, bbye..
> 2019-04-05 14:53:24,758 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler:
> Container container_e30_1553788280931_0013_01_01 completed with event
> FINISHED, but corresponding RMContainer doesn't exist.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9447) RM Crashes with NPE at TimelineServiceV2Publisher.putEntity
[
https://issues.apache.org/jira/browse/YARN-9447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817575#comment-16817575
]
Prabhu Joseph commented on YARN-9447:
-
Thanks [~yuan_zac]. Did not search existing jira's properly, will close this as
Duplicate.
> RM Crashes with NPE at TimelineServiceV2Publisher.putEntity
> ---
>
> Key: YARN-9447
> URL: https://issues.apache.org/jira/browse/YARN-9447
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Critical
> Attachments: YARN-9447-001.patch, rm.log
>
>
> ResourceManager crashes with NullPointerException when
> TimelineServiceV2Publisher does putEntity after the timeline collector
> service for application is removed. This happened when killing a mapreduce
> job.
> {code}
> 2019-04-05 14:53:24,728 INFO
> org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollectorManager:
> The collector service for application_1553788280931_0013 was removed
> 2019-04-05 14:53:24,734 FATAL org.apache.hadoop.yarn.event.AsyncDispatcher:
> Error in dispatcher thread
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher.putEntity(TimelineServiceV2Publisher.java:461)
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher.access$100(TimelineServiceV2Publisher.java:73)
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher$TimelineV2EventHandler.handle(TimelineServiceV2Publisher.java:496)
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher$TimelineV2EventHandler.handle(TimelineServiceV2Publisher.java:485)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:748)
> 2019-04-05 14:53:24,743 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Exiting, bbye..
> 2019-04-05 14:53:24,758 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler:
> Container container_e30_1553788280931_0013_01_01 completed with event
> FINISHED, but corresponding RMContainer doesn't exist.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9481) [JDK 11] Build fails due to hard-coded target version in hadoop-yarn-applications-catalog-webapp
[
https://issues.apache.org/jira/browse/YARN-9481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817544#comment-16817544
]
Hadoop QA commented on YARN-9481:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
29m 16s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 57s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
51s{color} | {color:green} hadoop-yarn-applications-catalog-webapp in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9481 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12965899/YARN-9481.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml |
| uname | Linux 910b4855ed8f 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 7a68e7a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/23952/testReport/ |
| Max. process+thread count | 414 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/23952/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> [JDK 11] Build fails due to hard-coded target version in
> hadoop-yarn-applications-catalog-webapp
>
>
> Key: YARN-9481
>
[jira] [Commented] (YARN-9470) Fix order of actual and expected expression in assert statements
[
https://issues.apache.org/jira/browse/YARN-9470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817542#comment-16817542
]
Steve Loughran commented on YARN-9470:
--
AssertJ is already on the classpath for hdfs-test; no reason not to use it in
YARN
> Fix order of actual and expected expression in assert statements
>
>
> Key: YARN-9470
> URL: https://issues.apache.org/jira/browse/YARN-9470
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9470-001.patch, assertEquals
>
>
> Fix order of actual and expected expression in assert statements which gives
> misleading message when test case fails. Attached file has some of the places
> where it is placed wrongly.
> {code}
> [ERROR]
> testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
> Time elapsed: 3.385 s <<< FAILURE!
> java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but
> was:<0>
> {code}
> For long term, [AssertJ|http://joel-costigliola.github.io/assertj/] can be
> used for new test cases which avoids such mistakes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9481) [JDK 11] Build fails due to hard-coded target version in hadoop-yarn-applications-catalog-webapp
[ https://issues.apache.org/jira/browse/YARN-9481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817536#comment-16817536 ] Hudson commented on YARN-9481: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16408 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16408/]) YARN-9481. [JDK 11] Build fails due to hard-coded target version in (aajisaka: rev c4c16cae7d813c979770b6d5f3245b91d0d2f728) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/pom.xml > [JDK 11] Build fails due to hard-coded target version in > hadoop-yarn-applications-catalog-webapp > > > Key: YARN-9481 > URL: https://issues.apache.org/jira/browse/YARN-9481 > Project: Hadoop YARN > Issue Type: Bug > Components: build >Affects Versions: 3.3.0 > Environment: hadoop 3.3.0 > jdk 11 >Reporter: Kei Kori >Assignee: Kei Kori >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9481.001.patch > > > source and target java versions in > hadoop-yarn-applications-catalog-webapp/pom.xml is hard-coded as 1.8. > It should be removed and delegated from parent pom.xml. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9481) [JDK 11] Build fails due to hard-coded target version in hadoop-yarn-applications-catalog-webapp
[ https://issues.apache.org/jira/browse/YARN-9481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated YARN-9481: Hadoop Flags: Reviewed Summary: [JDK 11] Build fails due to hard-coded target version in hadoop-yarn-applications-catalog-webapp (was: [JDK 11] Build with Java11 fails due to hard-coded javac.version in hadoop-yarn-applications-catalog-webapp) > [JDK 11] Build fails due to hard-coded target version in > hadoop-yarn-applications-catalog-webapp > > > Key: YARN-9481 > URL: https://issues.apache.org/jira/browse/YARN-9481 > Project: Hadoop YARN > Issue Type: Bug > Components: build >Affects Versions: 3.3.0 > Environment: hadoop 3.3.0 > jdk 11 >Reporter: Kei Kori >Assignee: Kei Kori >Priority: Major > Attachments: YARN-9481.001.patch > > > source and target java versions in > hadoop-yarn-applications-catalog-webapp/pom.xml is hard-coded as 1.8. > It should be removed and delegated from parent pom.xml. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9481) [JDK 11] Build with Java11 fails due to hard-coded javac.version in hadoop-yarn-applications-catalog-webapp
[ https://issues.apache.org/jira/browse/YARN-9481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817533#comment-16817533 ] Akira Ajisaka commented on YARN-9481: - +1, thank you for taking care of Java 11 environment. > [JDK 11] Build with Java11 fails due to hard-coded javac.version in > hadoop-yarn-applications-catalog-webapp > --- > > Key: YARN-9481 > URL: https://issues.apache.org/jira/browse/YARN-9481 > Project: Hadoop YARN > Issue Type: Bug > Components: build >Affects Versions: 3.3.0 > Environment: hadoop 3.3.0 > jdk 11 >Reporter: Kei Kori >Assignee: Kei Kori >Priority: Major > Attachments: YARN-9481.001.patch > > > source and target java versions in > hadoop-yarn-applications-catalog-webapp/pom.xml is hard-coded as 1.8. > It should be removed and delegated from parent pom.xml. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9481) [JDK 11] Build with Java11 fails due to hard-coded javac.version in hadoop-yarn-applications-catalog-webapp
[
https://issues.apache.org/jira/browse/YARN-9481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817530#comment-16817530
]
Hadoop QA commented on YARN-9481:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
29m 12s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 5s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
51s{color} | {color:green} hadoop-yarn-applications-catalog-webapp in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9481 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12965895/YARN-9481.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml |
| uname | Linux f66d7f3a0919 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 4b4fef2 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/23951/testReport/ |
| Max. process+thread count | 412 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/23951/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> [JDK 11] Build with Java11 fails due to hard-coded javac.version in
> hadoop-yarn-applications-catalog-webapp
> ---
>
> Key: YARN-9481
[jira] [Updated] (YARN-9481) [JDK 11] Build with Java11 fails due to hard-coded javac.version in hadoop-yarn-applications-catalog-webapp
[ https://issues.apache.org/jira/browse/YARN-9481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kei Kori updated YARN-9481: --- Attachment: (was: YARN-9481.patch) > [JDK 11] Build with Java11 fails due to hard-coded javac.version in > hadoop-yarn-applications-catalog-webapp > --- > > Key: YARN-9481 > URL: https://issues.apache.org/jira/browse/YARN-9481 > Project: Hadoop YARN > Issue Type: Bug > Components: build >Affects Versions: 3.3.0 > Environment: hadoop 3.3.0 > jdk 11 >Reporter: Kei Kori >Assignee: Kei Kori >Priority: Major > > source and target java versions in > hadoop-yarn-applications-catalog-webapp/pom.xml is hard-coded as 1.8. > It should be removed and delegated from parent pom.xml. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9474) Remove hard coded sleep from Opportunistic Scheduler tests.
[ https://issues.apache.org/jira/browse/YARN-9474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817521#comment-16817521 ] Hudson commented on YARN-9474: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16407 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16407/]) YARN-9474. Remove hard coded sleep from Opportunistic Scheduler tests. (inigoiri: rev 7a68e7abd51b9a0e25758e182eaa0a5cbbf13d87) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/scheduler/TestOpportunisticContainerAllocator.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestOpportunisticContainerAllocatorAMService.java > Remove hard coded sleep from Opportunistic Scheduler tests. > --- > > Key: YARN-9474 > URL: https://issues.apache.org/jira/browse/YARN-9474 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9474.001.patch, YARN-9474.002.patch > > > Remove hard coded sleep from Opportunistic Scheduler tests and improve logs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9474) Remove hard coded sleep from Opportunistic Scheduler tests.
[ https://issues.apache.org/jira/browse/YARN-9474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817517#comment-16817517 ] Abhishek Modi commented on YARN-9474: - Thanks [~elgoiri] for review and committing it. > Remove hard coded sleep from Opportunistic Scheduler tests. > --- > > Key: YARN-9474 > URL: https://issues.apache.org/jira/browse/YARN-9474 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9474.001.patch, YARN-9474.002.patch > > > Remove hard coded sleep from Opportunistic Scheduler tests and improve logs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9474) Remove hard coded sleep from Opportunistic Scheduler tests.
[ https://issues.apache.org/jira/browse/YARN-9474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817516#comment-16817516 ] Íñigo Goiri commented on YARN-9474: --- Thanks [~abmodi] for the patch. Committed to trunk. > Remove hard coded sleep from Opportunistic Scheduler tests. > --- > > Key: YARN-9474 > URL: https://issues.apache.org/jira/browse/YARN-9474 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9474.001.patch, YARN-9474.002.patch > > > Remove hard coded sleep from Opportunistic Scheduler tests and improve logs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9474) Remove hard coded sleep from Opportunistic Scheduler tests.
[ https://issues.apache.org/jira/browse/YARN-9474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated YARN-9474: -- Fix Version/s: 3.3.0 > Remove hard coded sleep from Opportunistic Scheduler tests. > --- > > Key: YARN-9474 > URL: https://issues.apache.org/jira/browse/YARN-9474 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9474.001.patch, YARN-9474.002.patch > > > Remove hard coded sleep from Opportunistic Scheduler tests and improve logs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9437) RMNodeImpls occupy too much memory and causes RM GC to take a long time
[ https://issues.apache.org/jira/browse/YARN-9437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qiuliang updated YARN-9437: --- Attachment: YARN-9437-v1.txt > RMNodeImpls occupy too much memory and causes RM GC to take a long time > --- > > Key: YARN-9437 > URL: https://issues.apache.org/jira/browse/YARN-9437 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.9.1 >Reporter: qiuliang >Priority: Minor > Attachments: 1.png, 2.png, 3.png, YARN-9437-v1.txt > > > We use hadoop-2.9.1 in our production environment with 1600+ nodes. 95.63% of > RM memory is occupied by RMNodeImpl. Analysis of RM memory found that each > RMNodeImpl has approximately 14M. The reason is that there is a 130,000+ > completedcontainers in each RMNodeImpl that has not been released. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9437) RMNodeImpls occupy too much memory and causes RM GC to take a long time
[ https://issues.apache.org/jira/browse/YARN-9437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qiuliang updated YARN-9437: --- Attachment: (was: YARN_9437-v1.txt) > RMNodeImpls occupy too much memory and causes RM GC to take a long time > --- > > Key: YARN-9437 > URL: https://issues.apache.org/jira/browse/YARN-9437 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.9.1 >Reporter: qiuliang >Priority: Minor > Attachments: 1.png, 2.png, 3.png > > > We use hadoop-2.9.1 in our production environment with 1600+ nodes. 95.63% of > RM memory is occupied by RMNodeImpl. Analysis of RM memory found that each > RMNodeImpl has approximately 14M. The reason is that there is a 130,000+ > completedcontainers in each RMNodeImpl that has not been released. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9437) RMNodeImpls occupy too much memory and causes RM GC to take a long time
[ https://issues.apache.org/jira/browse/YARN-9437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qiuliang updated YARN-9437: --- Attachment: YARN_9437-v1.txt > RMNodeImpls occupy too much memory and causes RM GC to take a long time > --- > > Key: YARN-9437 > URL: https://issues.apache.org/jira/browse/YARN-9437 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.9.1 >Reporter: qiuliang >Priority: Minor > Attachments: 1.png, 2.png, 3.png > > > We use hadoop-2.9.1 in our production environment with 1600+ nodes. 95.63% of > RM memory is occupied by RMNodeImpl. Analysis of RM memory found that each > RMNodeImpl has approximately 14M. The reason is that there is a 130,000+ > completedcontainers in each RMNodeImpl that has not been released. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9437) RMNodeImpls occupy too much memory and causes RM GC to take a long time
[
https://issues.apache.org/jira/browse/YARN-9437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817515#comment-16817515
]
Hadoop QA commented on YARN-9437:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} YARN-9437 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-9437 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/23950/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RMNodeImpls occupy too much memory and causes RM GC to take a long time
> ---
>
> Key: YARN-9437
> URL: https://issues.apache.org/jira/browse/YARN-9437
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.9.1
>Reporter: qiuliang
>Priority: Minor
> Attachments: 1.png, 2.png, 3.png
>
>
> We use hadoop-2.9.1 in our production environment with 1600+ nodes. 95.63% of
> RM memory is occupied by RMNodeImpl. Analysis of RM memory found that each
> RMNodeImpl has approximately 14M. The reason is that there is a 130,000+
> completedcontainers in each RMNodeImpl that has not been released.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9481) [JDK 11] Build with Java11 fails due to hard-coded javac.version in hadoop-yarn-applications-catalog-webapp
[ https://issues.apache.org/jira/browse/YARN-9481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kei Kori updated YARN-9481: --- Attachment: (was: YARN-9481.patch) > [JDK 11] Build with Java11 fails due to hard-coded javac.version in > hadoop-yarn-applications-catalog-webapp > --- > > Key: YARN-9481 > URL: https://issues.apache.org/jira/browse/YARN-9481 > Project: Hadoop YARN > Issue Type: Bug > Components: build >Affects Versions: 3.3.0 > Environment: hadoop 3.3.0 > jdk 11 >Reporter: Kei Kori >Assignee: Kei Kori >Priority: Major > Attachments: YARN-9481.patch > > > source and target java versions in > hadoop-yarn-applications-catalog-webapp/pom.xml is hard-coded as 1.8. > It should be removed and delegated from parent pom.xml. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9481) [JDK 11] Build with Java11 fails due to hard-coded javac.version in hadoop-yarn-applications-catalog-webapp
[ https://issues.apache.org/jira/browse/YARN-9481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kei Kori updated YARN-9481: --- Attachment: YARN-9481.patch > [JDK 11] Build with Java11 fails due to hard-coded javac.version in > hadoop-yarn-applications-catalog-webapp > --- > > Key: YARN-9481 > URL: https://issues.apache.org/jira/browse/YARN-9481 > Project: Hadoop YARN > Issue Type: Bug > Components: build >Affects Versions: 3.3.0 > Environment: hadoop 3.3.0 > jdk 11 >Reporter: Kei Kori >Assignee: Kei Kori >Priority: Major > Attachments: YARN-9481.patch > > > source and target java versions in > hadoop-yarn-applications-catalog-webapp/pom.xml is hard-coded as 1.8. > It should be removed and delegated from parent pom.xml. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9474) Remove hard coded sleep from Opportunistic Scheduler tests.
[ https://issues.apache.org/jira/browse/YARN-9474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817509#comment-16817509 ] Íñigo Goiri commented on YARN-9474: --- +1 on [^YARN-9474.002.patch]. Committing shortly. > Remove hard coded sleep from Opportunistic Scheduler tests. > --- > > Key: YARN-9474 > URL: https://issues.apache.org/jira/browse/YARN-9474 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-9474.001.patch, YARN-9474.002.patch > > > Remove hard coded sleep from Opportunistic Scheduler tests and improve logs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9447) RM Crashes with NPE at TimelineServiceV2Publisher.putEntity
[
https://issues.apache.org/jira/browse/YARN-9447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817505#comment-16817505
]
Zac Zhou commented on YARN-9447:
[~Prabhu Joseph],
Thanks for fixing it. I think this issue is the same as
[YARN-6695|https://issues.apache.org/jira/browse/YARN-6695]
> RM Crashes with NPE at TimelineServiceV2Publisher.putEntity
> ---
>
> Key: YARN-9447
> URL: https://issues.apache.org/jira/browse/YARN-9447
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Critical
> Attachments: YARN-9447-001.patch, rm.log
>
>
> ResourceManager crashes with NullPointerException when
> TimelineServiceV2Publisher does putEntity after the timeline collector
> service for application is removed. This happened when killing a mapreduce
> job.
> {code}
> 2019-04-05 14:53:24,728 INFO
> org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollectorManager:
> The collector service for application_1553788280931_0013 was removed
> 2019-04-05 14:53:24,734 FATAL org.apache.hadoop.yarn.event.AsyncDispatcher:
> Error in dispatcher thread
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher.putEntity(TimelineServiceV2Publisher.java:461)
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher.access$100(TimelineServiceV2Publisher.java:73)
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher$TimelineV2EventHandler.handle(TimelineServiceV2Publisher.java:496)
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher$TimelineV2EventHandler.handle(TimelineServiceV2Publisher.java:485)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:748)
> 2019-04-05 14:53:24,743 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Exiting, bbye..
> 2019-04-05 14:53:24,758 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler:
> Container container_e30_1553788280931_0013_01_01 completed with event
> FINISHED, but corresponding RMContainer doesn't exist.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9481) [JDK 11] Build with Java11 fails due to hard-coded javac.version in hadoop-yarn-applications-catalog-webapp
[ https://issues.apache.org/jira/browse/YARN-9481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated YARN-9481: Summary: [JDK 11] Build with Java11 fails due to hard-coded javac.version in hadoop-yarn-applications-catalog-webapp (was: Build with Java11 fails due to hard-coded javac.version in hadoop-yarn-applications-catalog-webapp) > [JDK 11] Build with Java11 fails due to hard-coded javac.version in > hadoop-yarn-applications-catalog-webapp > --- > > Key: YARN-9481 > URL: https://issues.apache.org/jira/browse/YARN-9481 > Project: Hadoop YARN > Issue Type: Bug > Components: build >Affects Versions: 3.3.0 > Environment: hadoop 3.3.0 > jdk 11 >Reporter: Kei Kori >Assignee: Kei Kori >Priority: Major > > source and target java versions in > hadoop-yarn-applications-catalog-webapp/pom.xml is hard-coded as 1.8. > It should be removed and delegated from parent pom.xml. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9481) Build with Java11 fails due to hard-coded javac.version in hadoop-yarn-applications-catalog-webapp
[ https://issues.apache.org/jira/browse/YARN-9481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka reassigned YARN-9481: --- Assignee: Kei Kori > Build with Java11 fails due to hard-coded javac.version in > hadoop-yarn-applications-catalog-webapp > -- > > Key: YARN-9481 > URL: https://issues.apache.org/jira/browse/YARN-9481 > Project: Hadoop YARN > Issue Type: Bug > Components: build >Affects Versions: 3.3.0 > Environment: hadoop 3.3.0 > jdk 11 >Reporter: Kei Kori >Assignee: Kei Kori >Priority: Major > > source and target java versions in > hadoop-yarn-applications-catalog-webapp/pom.xml is hard-coded as 1.8. > It should be removed and delegated from parent pom.xml. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9481) Build with Java11 fails due to hard-coded javac.version in hadoop-yarn-applications-catalog-webapp
Kei Kori created YARN-9481: -- Summary: Build with Java11 fails due to hard-coded javac.version in hadoop-yarn-applications-catalog-webapp Key: YARN-9481 URL: https://issues.apache.org/jira/browse/YARN-9481 Project: Hadoop YARN Issue Type: Bug Components: build Affects Versions: 3.3.0 Environment: hadoop 3.3.0 jdk 11 Reporter: Kei Kori source and target java versions in hadoop-yarn-applications-catalog-webapp/pom.xml is hard-coded as 1.8. It should be removed and delegated from parent pom.xml. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9379) Can't specify docker runtime through environment
[ https://issues.apache.org/jira/browse/YARN-9379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817372#comment-16817372 ] caozhiqiang edited comment on YARN-9379 at 4/14/19 5:51 PM: OK, I have resolve these indentation problems. Thank you very much, [~ebadger]. was (Author: caozhiqiang): OK, I have resolve these indentation problems. Thank you very much, [~ebadger] > Can't specify docker runtime through environment > > > Key: YARN-9379 > URL: https://issues.apache.org/jira/browse/YARN-9379 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.3.0 >Reporter: caozhiqiang >Assignee: caozhiqiang >Priority: Minor > Attachments: YARN-9379-branch-3.2.0.001.patch, YARN-9379.002.patch, > YARN-9379.003.patch, YARN-9379.004.patch, YARN-9379.005.patch, > YARN-9379.006.patch > > > When use docker to run yarn containers, even though there is > docker.allowed.runtimes in container-executor.cfg, there are not parameter to > specify the docker runtime, such as gvisor, lxc or kata. With this patch, > client can add parameter such as > -[Dyarn.app.mapreduce.am|http://dyarn.app.mapreduce.am/].env.YARN_CONTAINER_RUNTIME_DOCKER_RUNTIME=runsc > to specify docker runtime. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9379) Can't specify docker runtime through environment
[ https://issues.apache.org/jira/browse/YARN-9379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817372#comment-16817372 ] caozhiqiang commented on YARN-9379: --- OK, I have resolve these indentation problems. Thank you very much, [~ebadger] > Can't specify docker runtime through environment > > > Key: YARN-9379 > URL: https://issues.apache.org/jira/browse/YARN-9379 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.3.0 >Reporter: caozhiqiang >Assignee: caozhiqiang >Priority: Minor > Attachments: YARN-9379-branch-3.2.0.001.patch, YARN-9379.002.patch, > YARN-9379.003.patch, YARN-9379.004.patch, YARN-9379.005.patch, > YARN-9379.006.patch > > > When use docker to run yarn containers, even though there is > docker.allowed.runtimes in container-executor.cfg, there are not parameter to > specify the docker runtime, such as gvisor, lxc or kata. With this patch, > client can add parameter such as > -[Dyarn.app.mapreduce.am|http://dyarn.app.mapreduce.am/].env.YARN_CONTAINER_RUNTIME_DOCKER_RUNTIME=runsc > to specify docker runtime. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9470) Fix order of actual and expected expression in assert statements
[
https://issues.apache.org/jira/browse/YARN-9470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817339#comment-16817339
]
Hadoop QA commented on YARN-9470:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 64 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 8m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
20m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
19s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
in trunk has 2 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
46s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
45s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 44s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 2 new + 1935 unchanged - 15 fixed = 1937 total (was 1950) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 7m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 24s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 11m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
47s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
58s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m
38s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
59s{color} | {color:green} hadoop-yarn-server-web-proxy in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
22s{color} | {color:green}
[jira] [Updated] (YARN-9470) Fix order of actual and expected expression in assert statements
[
https://issues.apache.org/jira/browse/YARN-9470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated YARN-9470:
Attachment: (was: YARN-9470-001.patch)
> Fix order of actual and expected expression in assert statements
>
>
> Key: YARN-9470
> URL: https://issues.apache.org/jira/browse/YARN-9470
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9470-001.patch, assertEquals
>
>
> Fix order of actual and expected expression in assert statements which gives
> misleading message when test case fails. Attached file has some of the places
> where it is placed wrongly.
> {code}
> [ERROR]
> testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
> Time elapsed: 3.385 s <<< FAILURE!
> java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but
> was:<0>
> {code}
> For long term, [AssertJ|http://joel-costigliola.github.io/assertj/] can be
> used for new test cases which avoids such mistakes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9470) Fix order of actual and expected expression in assert statements
[
https://issues.apache.org/jira/browse/YARN-9470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated YARN-9470:
Attachment: YARN-9470-001.patch
> Fix order of actual and expected expression in assert statements
>
>
> Key: YARN-9470
> URL: https://issues.apache.org/jira/browse/YARN-9470
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9470-001.patch, assertEquals
>
>
> Fix order of actual and expected expression in assert statements which gives
> misleading message when test case fails. Attached file has some of the places
> where it is placed wrongly.
> {code}
> [ERROR]
> testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
> Time elapsed: 3.385 s <<< FAILURE!
> java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but
> was:<0>
> {code}
> For long term, [AssertJ|http://joel-costigliola.github.io/assertj/] can be
> used for new test cases which avoids such mistakes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9453) Clean up code long if-else chain in ApplicationCLI#run
[
https://issues.apache.org/jira/browse/YARN-9453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817272#comment-16817272
]
Hadoop QA commented on YARN-9453:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 21s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client:
The patch generated 0 new + 288 unchanged - 2 fixed = 288 total (was 290)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 13s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 25m 11s{color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 70m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.client.api.impl.TestAMRMClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9453 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12965872/YARN-9453.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 1dcaa9c9ccfa 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / b2cdf80 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/23948/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/23948/testReport/ |
| Max. process+thread count | 675 (vs. ulimit of 1) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U:
[jira] [Commented] (YARN-9453) Clean up code long if-else chain in ApplicationCLI#run
[ https://issues.apache.org/jira/browse/YARN-9453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817251#comment-16817251 ] Wanqiang Ji commented on YARN-9453: --- 003 patch fixed the checkstyle's issue. The UT failure is not related with this patch. Refer to YARN-9062 duplicate of YARN-6272 > Clean up code long if-else chain in ApplicationCLI#run > -- > > Key: YARN-9453 > URL: https://issues.apache.org/jira/browse/YARN-9453 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Wanqiang Ji >Priority: Major > Labels: newbie > Attachments: YARN-9453.001.patch, YARN-9453.002.patch, > YARN-9453.003.patch > > > org.apache.hadoop.yarn.client.cli.ApplicationCLI#run is 630 lines long, > contains a long if-else chain and many many conditions. > As a start, the bodies of the conditions could be extracted to methods and a > more clean solution could be introduced to parse the argument values. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9453) Clean up code long if-else chain in ApplicationCLI#run
[ https://issues.apache.org/jira/browse/YARN-9453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wanqiang Ji updated YARN-9453: -- Attachment: YARN-9453.003.patch > Clean up code long if-else chain in ApplicationCLI#run > -- > > Key: YARN-9453 > URL: https://issues.apache.org/jira/browse/YARN-9453 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Wanqiang Ji >Priority: Major > Labels: newbie > Attachments: YARN-9453.001.patch, YARN-9453.002.patch, > YARN-9453.003.patch > > > org.apache.hadoop.yarn.client.cli.ApplicationCLI#run is 630 lines long, > contains a long if-else chain and many many conditions. > As a start, the bodies of the conditions could be extracted to methods and a > more clean solution could be introduced to parse the argument values. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9453) Clean up code long if-else chain in ApplicationCLI#run
[
https://issues.apache.org/jira/browse/YARN-9453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817235#comment-16817235
]
Hadoop QA commented on YARN-9453:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 58s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 21s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client: The patch generated 4 new +
288 unchanged - 2 fixed = 292 total (was 290) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 16s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 26m 23s{color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 73m 35s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.client.api.impl.TestAMRMClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9453 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12965860/YARN-9453.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 63c3453c1657 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / b2cdf80 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/23947/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/23947/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/23947/testReport/ |
| Max.
[jira] [Created] (YARN-9480) createAppDir() in LogAggregationService shouldn't block dispatcher thread of ContainerManagerImpl
liyakun created YARN-9480: - Summary: createAppDir() in LogAggregationService shouldn't block dispatcher thread of ContainerManagerImpl Key: YARN-9480 URL: https://issues.apache.org/jira/browse/YARN-9480 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Reporter: liyakun Assignee: liyakun At present, when startContainers(), if NM does not contain the application, it will enter the step of INIT_APPLICATION. In the application init step, createAppDir() will be executed, and it is a blocking operation. createAppDir() is an operation that needs to interact with an external file system. This operation is affected by the SLA of the external file system. Once the external file system has a high latency, the NM dispatcher thread of ContainerManagerImpl will be stuck. (In fact, I have seen a scene that NM stuck here for more than an hour.) I think it would be more reasonable to move createAppDir() to the actual time of uploading log (in other threads). And according to the logRetentionPolicy, many of the containers may not get to this step, which will save a lot of interactions with external file system. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9453) Clean up code long if-else chain in ApplicationCLI#run
[ https://issues.apache.org/jira/browse/YARN-9453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817210#comment-16817210 ] Wanqiang Ji commented on YARN-9453: --- 002 patch fixed all the issues from Yetus. > Clean up code long if-else chain in ApplicationCLI#run > -- > > Key: YARN-9453 > URL: https://issues.apache.org/jira/browse/YARN-9453 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Wanqiang Ji >Priority: Major > Labels: newbie > Attachments: YARN-9453.001.patch, YARN-9453.002.patch > > > org.apache.hadoop.yarn.client.cli.ApplicationCLI#run is 630 lines long, > contains a long if-else chain and many many conditions. > As a start, the bodies of the conditions could be extracted to methods and a > more clean solution could be introduced to parse the argument values. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9453) Clean up code long if-else chain in ApplicationCLI#run
[ https://issues.apache.org/jira/browse/YARN-9453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wanqiang Ji updated YARN-9453: -- Attachment: YARN-9453.002.patch > Clean up code long if-else chain in ApplicationCLI#run > -- > > Key: YARN-9453 > URL: https://issues.apache.org/jira/browse/YARN-9453 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Wanqiang Ji >Priority: Major > Labels: newbie > Attachments: YARN-9453.001.patch, YARN-9453.002.patch > > > org.apache.hadoop.yarn.client.cli.ApplicationCLI#run is 630 lines long, > contains a long if-else chain and many many conditions. > As a start, the bodies of the conditions could be extracted to methods and a > more clean solution could be introduced to parse the argument values. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9379) Can't specify docker runtime through environment
[
https://issues.apache.org/jira/browse/YARN-9379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817201#comment-16817201
]
Hadoop QA commented on YARN-9379:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 25s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
10s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 42s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
54s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
58s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m
29s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
45s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}110m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9379 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12965848/YARN-9379.006.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 945617ae9c4f 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
