date:20190130



[ 
https://issues.apache.org/jira/browse/YARN-9161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756976#comment-16756976
 ] 

Hadoop QA commented on YARN-9161:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 7 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  3m  
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
17m 54s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m  
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
44s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m 
55s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m 43s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 2 new + 387 unchanged - 6 fixed = 389 total (was 393) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  3m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
15m 23s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
47s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
40s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 95m 34s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 16m 
11s{color} | {color:green} hadoop-yarn-applications-distributedshell in the 
patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
39s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}218m 46s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.resourcemanager.rmapp.attempt.TestRMAppAttemptTransitions |
|   | hadoop.yarn.server.resourcemanager.TestCapacitySchedulerMetrics |
|   | 
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerSurgicalPreemption
 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9161 |
| JIRA Patch URL |

[jira] [Comment Edited] (YARN-9250) hadoop-yarn-server-nodemanager build failed: make failed with error code 2



[ 
https://issues.apache.org/jira/browse/YARN-9250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756958#comment-16756958
 ] 

charlie mao edited comment on YARN-9250 at 1/31/19 7:31 AM:


 there is a nullpointerException error.
 am i need to modify the pom.xml in hadoop-yarn-server-nodemanager sub project?


was (Author: linlong):
 there is a nullpointerException error.
 am i need to modify the pom.xml in hadoop-yarn-server-nodemanager project?

> hadoop-yarn-server-nodemanager build failed: make failed with error code 2
> --
>
> Key: YARN-9250
> URL: https://issues.apache.org/jira/browse/YARN-9250
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 3.2.0
>Reporter: charlie mao
>Priority: Blocker
>
> when i compile hadoop-3.2.0 release,i encountered the following errors：
> [ERROR] Failed to execute goal 
> org.apache.hadoop:hadoop-maven-plugins:3.2.0:cmake-compile (cmake-compile) on 
> project hadoop-yarn-server-nodemanager: make failed with error code 2 -> 
> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
> goal org.apache.hadoop:hadoop-maven-plugins:3.2.0:cmake-compile 
> (cmake-compile) on project hadoop-yarn-server-nodemanager: make failed with 
> error code 2
>  at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
>  at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>  at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>  at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
>  at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
>  at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>  at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
>  at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
>  at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
>  at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
>  at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
>  at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
>  at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>  at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498)
>  at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
>  at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
>  at 
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
>  at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
> Caused by: org.apache.maven.plugin.MojoExecutionException: make failed with 
> error code 2
>  at 
> org.apache.hadoop.maven.plugin.cmakebuilder.CompileMojo.runMake(CompileMojo.java:231)
>  at 
> org.apache.hadoop.maven.plugin.cmakebuilder.CompileMojo.execute(CompileMojo.java:98)
>  at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
>  at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
>  ... 20 more
> [ERROR]
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please 
> read the following articles:
> [ERROR] [Help 1] 
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the 
> command
> [ERROR] mvn  -rf :hadoop-yarn-server-nodemanager
>  
> my compiling environment：
> jdk 1.8.0_181
> maven:3.3.9(/3.6.0)
> cmake version 3.12.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9250) hadoop-yarn-server-nodemanager build failed: make failed with error code 2



[ 
https://issues.apache.org/jira/browse/YARN-9250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756958#comment-16756958
 ] 

charlie mao commented on YARN-9250:
---

 there is a nullpointerException error.
 am i need to modify the pom.xml in hadoop-yarn-server-nodemanager project?

> hadoop-yarn-server-nodemanager build failed: make failed with error code 2
> --
>
> Key: YARN-9250
> URL: https://issues.apache.org/jira/browse/YARN-9250
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 3.2.0
>Reporter: charlie mao
>Priority: Blocker
>
> when i compile hadoop-3.2.0 release,i encountered the following errors：
> [ERROR] Failed to execute goal 
> org.apache.hadoop:hadoop-maven-plugins:3.2.0:cmake-compile (cmake-compile) on 
> project hadoop-yarn-server-nodemanager: make failed with error code 2 -> 
> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
> goal org.apache.hadoop:hadoop-maven-plugins:3.2.0:cmake-compile 
> (cmake-compile) on project hadoop-yarn-server-nodemanager: make failed with 
> error code 2
>  at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
>  at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>  at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>  at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
>  at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
>  at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>  at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
>  at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
>  at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
>  at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
>  at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
>  at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
>  at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>  at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498)
>  at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
>  at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
>  at 
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
>  at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
> Caused by: org.apache.maven.plugin.MojoExecutionException: make failed with 
> error code 2
>  at 
> org.apache.hadoop.maven.plugin.cmakebuilder.CompileMojo.runMake(CompileMojo.java:231)
>  at 
> org.apache.hadoop.maven.plugin.cmakebuilder.CompileMojo.execute(CompileMojo.java:98)
>  at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
>  at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
>  ... 20 more
> [ERROR]
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please 
> read the following articles:
> [ERROR] [Help 1] 
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the 
> command
> [ERROR] mvn  -rf :hadoop-yarn-server-nodemanager
>  
> my compiling environment：
> jdk 1.8.0_181
> maven:3.3.9(/3.6.0)
> cmake version 3.12.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9250) hadoop-yarn-server-nodemanager build failed: make failed with error code 2



[ 
https://issues.apache.org/jira/browse/YARN-9250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756954#comment-16756954
 ] 

charlie mao commented on YARN-9250:
---

thanks for reply!

yes, i've tried to  compile hadoop-3.2.0 release three times, but all failed.
i did success in compiling hadoop-3.1.0 before long. 
please take a look at the following error:

[ERROR] Failed to execute goal 
org.apache.hadoop:hadoop-maven-plugins:3.2.0:cmake-compile (default-cli) on 
project hadoop-yarn-server-nodemanager: Execution default-cli of goal 
org.apache.hadoop:hadoop-maven-plugins:3.2.0:cmake-compile failed.: 
NullPointerException -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal 
org.apache.hadoop:hadoop-maven-plugins:3.2.0:cmake-compile (default-cli) on 
project hadoop-yarn-server-nodemanager: Execution default-cli of goal 
org.apache.hadoop:hadoop-maven-plugins:3.2.0:cmake-compile failed.
at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:215)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
(LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
(LifecycleModuleBuilder.java:81)
at 
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
 (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute 
(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke 
(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke 
(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced 
(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch 
(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode 
(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main 
(Launcher.java:356)
Caused by: org.apache.maven.plugin.PluginExecutionException: Execution 
default-cli of goal org.apache.hadoop:hadoop-maven-plugins:3.2.0:cmake-compile 
failed.
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo 
(DefaultBuildPluginManager.java:148)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:210)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
(LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
(LifecycleModuleBuilder.java:81)
at 
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
 (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute 
(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke 
(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke 
(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced 
(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch 
(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode 
(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main 
(Launcher.java:356)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.maven.plugin.cmakebuilder.CompileMojo.runCMake 
(CompileMojo.java:143)
at

[jira] [Updated] (YARN-9257) Distributed Shell throws a NPE for a non-existent queue



 [ 
https://issues.apache.org/jira/browse/YARN-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Charan Hebri updated YARN-9257:
---
Description: 
When a non-existent queue is specified as a param for launching a distributed 
shell, a NPE is thrown.
{noformat}
19/01/30 09:41:26 ERROR distributedshell.Client: Error running Client
java.lang.NullPointerException
at 
org.apache.hadoop.yarn.applications.distributedshell.Client.run(Client.java:652)
at 
org.apache.hadoop.yarn.applications.distributedshell.Client.main(Client.java:265){noformat}

  was:
When a non-existent queue is specified as a param for launching a distributed 
shell, a NPE is thrown.
{noformat}
19/01/30 09:41:26 ERROR distributedshell.Client: Error running Client

java.lang.NullPointerException

at 
org.apache.hadoop.yarn.applications.distributedshell.Client.run(Client.java:652)

at 
org.apache.hadoop.yarn.applications.distributedshell.Client.main(Client.java:265){noformat}


> Distributed Shell throws a NPE for a non-existent queue
> ---
>
> Key: YARN-9257
> URL: https://issues.apache.org/jira/browse/YARN-9257
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: distributed-shell
>Reporter: Charan Hebri
>Assignee: Charan Hebri
>Priority: Major
>
> When a non-existent queue is specified as a param for launching a distributed 
> shell, a NPE is thrown.
> {noformat}
> 19/01/30 09:41:26 ERROR distributedshell.Client: Error running Client
> java.lang.NullPointerException
> at 
> org.apache.hadoop.yarn.applications.distributedshell.Client.run(Client.java:652)
> at 
> org.apache.hadoop.yarn.applications.distributedshell.Client.main(Client.java:265){noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9257) Distributed Shell client throws a NPE for a non-existent queue



[ 
https://issues.apache.org/jira/browse/YARN-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756934#comment-16756934
 ] 

Sunil Govindan commented on YARN-9257:
--

Change seems fine. Lets wait for jenkins.

> Distributed Shell client throws a NPE for a non-existent queue
> --
>
> Key: YARN-9257
> URL: https://issues.apache.org/jira/browse/YARN-9257
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: distributed-shell
>Reporter: Charan Hebri
>Assignee: Charan Hebri
>Priority: Major
> Attachments: YARN-9257.01.patch
>
>
> When a non-existent queue is specified as a param for launching a distributed 
> shell, a NPE is thrown.
> {noformat}
> 19/01/30 09:41:26 ERROR distributedshell.Client: Error running Client
> java.lang.NullPointerException
> at 
> org.apache.hadoop.yarn.applications.distributedshell.Client.run(Client.java:652)
> at 
> org.apache.hadoop.yarn.applications.distributedshell.Client.main(Client.java:265){noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9250) hadoop-yarn-server-nodemanager build failed: make failed with error code 2



[ 
https://issues.apache.org/jira/browse/YARN-9250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756935#comment-16756935
 ] 

Sunil Govindan commented on YARN-9250:
--

Hi [~linlong], are u able to compile trunk code or similar. 
to me, this looks like an environment issue in you mac.

> hadoop-yarn-server-nodemanager build failed: make failed with error code 2
> --
>
> Key: YARN-9250
> URL: https://issues.apache.org/jira/browse/YARN-9250
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 3.2.0
>Reporter: charlie mao
>Priority: Blocker
>
> when i compile hadoop-3.2.0 release,i encountered the following errors：
> [ERROR] Failed to execute goal 
> org.apache.hadoop:hadoop-maven-plugins:3.2.0:cmake-compile (cmake-compile) on 
> project hadoop-yarn-server-nodemanager: make failed with error code 2 -> 
> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
> goal org.apache.hadoop:hadoop-maven-plugins:3.2.0:cmake-compile 
> (cmake-compile) on project hadoop-yarn-server-nodemanager: make failed with 
> error code 2
>  at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
>  at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>  at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>  at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
>  at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
>  at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>  at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
>  at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
>  at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
>  at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
>  at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
>  at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
>  at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>  at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498)
>  at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
>  at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
>  at 
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
>  at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
> Caused by: org.apache.maven.plugin.MojoExecutionException: make failed with 
> error code 2
>  at 
> org.apache.hadoop.maven.plugin.cmakebuilder.CompileMojo.runMake(CompileMojo.java:231)
>  at 
> org.apache.hadoop.maven.plugin.cmakebuilder.CompileMojo.execute(CompileMojo.java:98)
>  at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
>  at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
>  ... 20 more
> [ERROR]
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please 
> read the following articles:
> [ERROR] [Help 1] 
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the 
> command
> [ERROR] mvn  -rf :hadoop-yarn-server-nodemanager
>  
> my compiling environment：
> jdk 1.8.0_181
> maven:3.3.9(/3.6.0)
> cmake version 3.12.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9257) Distributed Shell client throws a NPE for a non-existent queue



[ 
https://issues.apache.org/jira/browse/YARN-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756932#comment-16756932
 ] 

Charan Hebri commented on YARN-9257:


cc [~sunilg]

> Distributed Shell client throws a NPE for a non-existent queue
> --
>
> Key: YARN-9257
> URL: https://issues.apache.org/jira/browse/YARN-9257
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: distributed-shell
>Reporter: Charan Hebri
>Assignee: Charan Hebri
>Priority: Major
> Attachments: YARN-9257.01.patch
>
>
> When a non-existent queue is specified as a param for launching a distributed 
> shell, a NPE is thrown.
> {noformat}
> 19/01/30 09:41:26 ERROR distributedshell.Client: Error running Client
> java.lang.NullPointerException
> at 
> org.apache.hadoop.yarn.applications.distributedshell.Client.run(Client.java:652)
> at 
> org.apache.hadoop.yarn.applications.distributedshell.Client.main(Client.java:265){noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9257) Distributed Shell client throws a NPE for a non-existent queue



 [ 
https://issues.apache.org/jira/browse/YARN-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Charan Hebri updated YARN-9257:
---
Summary: Distributed Shell client throws a NPE for a non-existent queue  
(was: Distributed Shell throws a NPE for a non-existent queue)

> Distributed Shell client throws a NPE for a non-existent queue
> --
>
> Key: YARN-9257
> URL: https://issues.apache.org/jira/browse/YARN-9257
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: distributed-shell
>Reporter: Charan Hebri
>Assignee: Charan Hebri
>Priority: Major
> Attachments: YARN-9257.01.patch
>
>
> When a non-existent queue is specified as a param for launching a distributed 
> shell, a NPE is thrown.
> {noformat}
> 19/01/30 09:41:26 ERROR distributedshell.Client: Error running Client
> java.lang.NullPointerException
> at 
> org.apache.hadoop.yarn.applications.distributedshell.Client.run(Client.java:652)
> at 
> org.apache.hadoop.yarn.applications.distributedshell.Client.main(Client.java:265){noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9257) Distributed Shell throws a NPE for a non-existent queue



 [ 
https://issues.apache.org/jira/browse/YARN-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Charan Hebri updated YARN-9257:
---
Attachment: YARN-9257.01.patch

> Distributed Shell throws a NPE for a non-existent queue
> ---
>
> Key: YARN-9257
> URL: https://issues.apache.org/jira/browse/YARN-9257
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: distributed-shell
>Reporter: Charan Hebri
>Assignee: Charan Hebri
>Priority: Major
> Attachments: YARN-9257.01.patch
>
>
> When a non-existent queue is specified as a param for launching a distributed 
> shell, a NPE is thrown.
> {noformat}
> 19/01/30 09:41:26 ERROR distributedshell.Client: Error running Client
> java.lang.NullPointerException
> at 
> org.apache.hadoop.yarn.applications.distributedshell.Client.run(Client.java:652)
> at 
> org.apache.hadoop.yarn.applications.distributedshell.Client.main(Client.java:265){noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-9257) Distributed Shell throws a NPE for a non-existent queue

Charan Hebri created YARN-9257:
--

 Summary: Distributed Shell throws a NPE for a non-existent queue
 Key: YARN-9257
 URL: https://issues.apache.org/jira/browse/YARN-9257
 Project: Hadoop YARN
  Issue Type: Bug
  Components: distributed-shell
Reporter: Charan Hebri
Assignee: Charan Hebri


When a non-existent queue is specified as a param for launching a distributed 
shell, a NPE is thrown.
{noformat}
19/01/30 09:41:26 ERROR distributedshell.Client: Error running Client

java.lang.NullPointerException

at 
org.apache.hadoop.yarn.applications.distributedshell.Client.run(Client.java:652)

at 
org.apache.hadoop.yarn.applications.distributedshell.Client.main(Client.java:265){noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9250) hadoop-yarn-server-nodemanager build failed: make failed with error code 2



 [ 
https://issues.apache.org/jira/browse/YARN-9250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

charlie mao updated YARN-9250:
--
Description: 
when i compile hadoop-3.2.0 release,i encountered the following errors：

[ERROR] Failed to execute goal 
org.apache.hadoop:hadoop-maven-plugins:3.2.0:cmake-compile (cmake-compile) on 
project hadoop-yarn-server-nodemanager: make failed with error code 2 -> [Help 
1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal 
org.apache.hadoop:hadoop-maven-plugins:3.2.0:cmake-compile (cmake-compile) on 
project hadoop-yarn-server-nodemanager: make failed with error code 2
 at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
 at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
 at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
 at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
 at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
 at 
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
 at 
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
 at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
 at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
 at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
 at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
 at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
 at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at 
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
 at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
 at 
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
 at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: make failed with 
error code 2
 at 
org.apache.hadoop.maven.plugin.cmakebuilder.CompileMojo.runMake(CompileMojo.java:231)
 at 
org.apache.hadoop.maven.plugin.cmakebuilder.CompileMojo.execute(CompileMojo.java:98)
 at 
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
 at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
 ... 20 more
[ERROR]
[ERROR]
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn  -rf :hadoop-yarn-server-nodemanager

 

my compiling environment：

jdk 1.8.0_181

maven:3.3.9(/3.6.0)

cmake version 3.12.0



  was:
when i compile hadoop-3.2.0 release,i encountered the following errors：

[ERROR] Failed to execute goal 
org.apache.hadoop:hadoop-maven-plugins:3.2.0:cmake-compile (cmake-compile) on 
project hadoop-yarn-server-nodemanager: make failed with error code 2 -> [Help 
1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal 
org.apache.hadoop:hadoop-maven-plugins:3.2.0:cmake-compile (cmake-compile) on 
project hadoop-yarn-server-nodemanager: make failed with error code 2
 at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
 at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
 at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
 at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
 at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
 at 
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
 at 
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
 at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
 at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
 at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
 at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
 at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
 at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at

[jira] [Commented] (YARN-9208) Distributed shell allow LocalResourceVisibility to be specified



[ 
https://issues.apache.org/jira/browse/YARN-9208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756927#comment-16756927
 ] 

Prabhu Joseph commented on YARN-9208:
-

[~bibinchundatt] Thanks for the review, working on it. Will update you.

> Distributed shell allow LocalResourceVisibility to be specified
> ---
>
> Key: YARN-9208
> URL: https://issues.apache.org/jira/browse/YARN-9208
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Bibin A Chundatt
>Assignee: Prabhu Joseph
>Priority: Minor
> Attachments: YARN-9208-001.patch
>
>
> YARN-9008 add feature to add list of files to be localized.
> Would be great to have Visibility type too. Allows testing of PRIVATE and 
> PUBLIC type too



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9256) Make ATSv2 compilation default with hbase.profile=2.0



[ 
https://issues.apache.org/jira/browse/YARN-9256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756922#comment-16756922
 ] 

Hadoop QA commented on YARN-9256:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m 
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m 
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
56m 53s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
14s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 14m 18s{color} 
| {color:red} root generated 170 new + 1496 unchanged - 0 fixed = 1666 total 
(was 1496) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 40s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
22s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
31s{color} | {color:green} hadoop-yarn-server-timelineservice-hbase-server in 
the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  4m 45s{color} 
| {color:red} hadoop-yarn-server-timelineservice-hbase-tests in the patch 
failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
39s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 98m 24s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageSchema |
|   | hadoop.yarn.server.timelineservice.storage.TestTimelineReaderHBaseDown |
|   | 
hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
 |
|   | 
hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities |
|   | 
hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction
 |
|   | 
hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity |
|   | hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps |
|   | hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun |
|   | hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageDomain 
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce

[jira] [Comment Edited] (YARN-9060) [YARN-8851] Phase 1 - Support device isolation and use the Nvidia GPU plugin as an example



[ 
https://issues.apache.org/jira/browse/YARN-9060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756896#comment-16756896
 ] 

Sunil Govindan edited comment on YARN-9060 at 1/31/19 6:01 AM:
---

Thanks [~tangzhankun] 

Few comments.
 # In below comments, it's better to have majorNumber minorNumber example given 
in same line rather than giving in a line above.
{code:java}
# #for instance, "8:16,8:32"
# devices.denied-numbers=## Blacklisted devices not permitted to use. The 
format is comma separated "majorNumber:minorNumber". Leave it empty means 
default devices reported by device plugin are all allowed.
\ No newline at end of file{code}
 # In {{getRegisterRequestInfo}} , could we set the "nvidia.com/gpu" as a 
static value. ?
 # Its better to place NvidiaCommandExecutor under test package.
 # Thanks for adding detailed logs. it certainly helpful. Could you also pls 
improve some logs in updateDockerRunCommand to add container id or other 
dynamic values as well. pls refer below logs which lacks some more dynamic 
information. Also pls see other methods as well to check the same.
{code:java}
LOG.warn("The container return null for device runtime spec");
LOG.debug("Try to update docker run command.");{code}
 # requestsDevice ==> requestedDevice
 # Is there any chance that getCleanupDockerVolumesCommand wont be called in 
some failure case, if there is such chance, i worry about the cached data 
structures which may cause some leaks.
 # In getDeviceType, i prefer to keep a new enum for more readability than 
keeping "c" or "b"


was (Author: sunilg):
Thanks [~tangzhankun] 

Few comments.
 # In below comments, it's better to have majorNumber minorNumber example given 
in same line rather than giving in a line above.
{code:java}
# #for instance, "8:16,8:32"
# devices.denied-numbers=## Blacklisted devices not permitted to use. The 
format is comma separated "majorNumber:minorNumber". Leave it empty means 
default devices reported by device plugin are all allowed.
\ No newline at end of file{code}

 # In {{getRegisterRequestInfo}} , could we set the "nvidia.com/gpu" as a 
static value. ?
 # Its better to place NvidiaCommandExecutor under test package.
 # Thanks for adding detailed logs. it certainly helpful. Could you also pls 
improve some logs in updateDockerRunCommand to add container id or other 
dynamic values as well. pls refer below logs which lacks some more dynamic 
information. Also pls see other methods as well to check the same.
{code:java}
LOG.warn("The container return null for device runtime spec");
LOG.debug("Try to update docker run command.");{code}
 # requestsDevice ==> requestedDevice
 # Is there any chance that getCleanupDockerVolumesCommand wont be called in 
some failure case, if there is such chance, i worry about the cached data 
structures which may cause some leaks.
 # In getDeviceType, i prefer to keep a new enum for more readability than 
keeping "c" or "b"

> [YARN-8851] Phase 1 - Support device isolation and use the Nvidia GPU plugin 
> as an example
> --
>
> Key: YARN-9060
> URL: https://issues.apache.org/jira/browse/YARN-9060
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
>Priority: Major
> Attachments: YARN-9060-trunk.001.patch, YARN-9060-trunk.002.patch, 
> YARN-9060-trunk.003.patch, YARN-9060-trunk.004.patch, 
> YARN-9060-trunk.005.patch, YARN-9060-trunk.006.patch, 
> YARN-9060-trunk.007.patch, YARN-9060-trunk.008.patch, 
> YARN-9060-trunk.009.patch, YARN-9060-trunk.010.patch, 
> YARN-9060-trunk.011.patch, YARN-9060-trunk.012.patch, 
> YARN-9060-trunk.013.patch, YARN-9060-trunk.014.patch
>
>
> Due to the cgroups v1 implementation policy in linux kernel, we cannot update 
> the value of the device cgroups controller unless we have the root permission 
> ([here|https://github.com/torvalds/linux/blob/6f0d349d922ba44e4348a17a78ea51b7135965b1/security/device_cgroup.c#L604]).
>  So we need to support this in container-executor for Java layer to invoke.
> This Jira will have three parts:
>  # native c-e module
>  # Java layer code to isolate devices for container (docker and non-docker)
>  # A sample Nvidia GPU plugin



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-9060) [YARN-8851] Phase 1 - Support device isolation and use the Nvidia GPU plugin as an example



[ 
https://issues.apache.org/jira/browse/YARN-9060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756896#comment-16756896
 ] 

Sunil Govindan edited comment on YARN-9060 at 1/31/19 6:01 AM:
---

Thanks [~tangzhankun] 

Few comments.
 # In below comments, it's better to have majorNumber minorNumber example given 
in same line rather than giving in a line above.
{code:java}
# #for instance, "8:16,8:32"
# devices.denied-numbers=## Blacklisted devices not permitted to use. The 
format is comma separated "majorNumber:minorNumber". Leave it empty means 
default devices reported by device plugin are all allowed.
\ No newline at end of file{code}

 # In {{getRegisterRequestInfo}} , could we set the "nvidia.com/gpu" as a 
static value. ?
 # Its better to place NvidiaCommandExecutor under test package.
 # Thanks for adding detailed logs. it certainly helpful. Could you also pls 
improve some logs in updateDockerRunCommand to add container id or other 
dynamic values as well. pls refer below logs which lacks some more dynamic 
information. Also pls see other methods as well to check the same.
{code:java}
LOG.warn("The container return null for device runtime spec");
LOG.debug("Try to update docker run command.");{code}
 # requestsDevice ==> requestedDevice
 # Is there any chance that getCleanupDockerVolumesCommand wont be called in 
some failure case, if there is such chance, i worry about the cached data 
structures which may cause some leaks.
 # In getDeviceType, i prefer to keep a new enum for more readability than 
keeping "c" or "b"


was (Author: sunilg):
Thanks [~tangzhankun] 

Few comments.
 # In below comments, it's better to have majorNumber minorNumber example given 
in same line rather than giving in a line above.
{code:java}
# #for instance, "8:16,8:32"
# devices.denied-numbers=## Blacklisted devices not permitted to use. The 
format is comma separated "majorNumber:minorNumber". Leave it empty means 
default devices reported by device plugin are all allowed.
\ No newline at end of file{code}

 # In {{getRegisterRequestInfo}} , could we set the "nvidia.com/gpu" as a 
static value. ?
 # Its better to place NvidiaCommandExecutor under test package.
 # Thanks for adding detailed logs. it certainly helpful. Could you also pls 
improve some logs in updateDockerRunCommand to add container id or other 
dynamic values as well. pls refer below logs which lacks some more dynamic 
information. Also pls see other methods as well to check the same.
{code:java}
LOG.warn("The container return null for device runtime spec");
LOG.debug("Try to update docker run command.");{code}

 # requestsDevice ==> requestedDevice
 # Is there any chance that getCleanupDockerVolumesCommand wont be called in 
some failure case, if there is such chance, i worry about the cached data 
structures which may cause some leaks.
 # In getDeviceType, i prefer to keep a new enum for more readability than 
keeping "c" or "b"

> [YARN-8851] Phase 1 - Support device isolation and use the Nvidia GPU plugin 
> as an example
> --
>
> Key: YARN-9060
> URL: https://issues.apache.org/jira/browse/YARN-9060
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
>Priority: Major
> Attachments: YARN-9060-trunk.001.patch, YARN-9060-trunk.002.patch, 
> YARN-9060-trunk.003.patch, YARN-9060-trunk.004.patch, 
> YARN-9060-trunk.005.patch, YARN-9060-trunk.006.patch, 
> YARN-9060-trunk.007.patch, YARN-9060-trunk.008.patch, 
> YARN-9060-trunk.009.patch, YARN-9060-trunk.010.patch, 
> YARN-9060-trunk.011.patch, YARN-9060-trunk.012.patch, 
> YARN-9060-trunk.013.patch, YARN-9060-trunk.014.patch
>
>
> Due to the cgroups v1 implementation policy in linux kernel, we cannot update 
> the value of the device cgroups controller unless we have the root permission 
> ([here|https://github.com/torvalds/linux/blob/6f0d349d922ba44e4348a17a78ea51b7135965b1/security/device_cgroup.c#L604]).
>  So we need to support this in container-executor for Java layer to invoke.
> This Jira will have three parts:
>  # native c-e module
>  # Java layer code to isolate devices for container (docker and non-docker)
>  # A sample Nvidia GPU plugin



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9060) [YARN-8851] Phase 1 - Support device isolation and use the Nvidia GPU plugin as an example



[ 
https://issues.apache.org/jira/browse/YARN-9060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756896#comment-16756896
 ] 

Sunil Govindan commented on YARN-9060:
--

Thanks [~tangzhankun] 

Few comments.
 # In below comments, it's better to have majorNumber minorNumber example given 
in same line rather than giving in a line above.
{code:java}
# #for instance, "8:16,8:32"
# devices.denied-numbers=## Blacklisted devices not permitted to use. The 
format is comma separated "majorNumber:minorNumber". Leave it empty means 
default devices reported by device plugin are all allowed.
\ No newline at end of file{code}

 # In {{getRegisterRequestInfo}} , could we set the "nvidia.com/gpu" as a 
static value. ?
 # Its better to place NvidiaCommandExecutor under test package.
 # Thanks for adding detailed logs. it certainly helpful. Could you also pls 
improve some logs in updateDockerRunCommand to add container id or other 
dynamic values as well. pls refer below logs which lacks some more dynamic 
information. Also pls see other methods as well to check the same.
{code:java}
LOG.warn("The container return null for device runtime spec");
LOG.debug("Try to update docker run command.");{code}

 # requestsDevice ==> requestedDevice
 # Is there any chance that getCleanupDockerVolumesCommand wont be called in 
some failure case, if there is such chance, i worry about the cached data 
structures which may cause some leaks.
 # In getDeviceType, i prefer to keep a new enum for more readability than 
keeping "c" or "b"

> [YARN-8851] Phase 1 - Support device isolation and use the Nvidia GPU plugin 
> as an example
> --
>
> Key: YARN-9060
> URL: https://issues.apache.org/jira/browse/YARN-9060
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
>Priority: Major
> Attachments: YARN-9060-trunk.001.patch, YARN-9060-trunk.002.patch, 
> YARN-9060-trunk.003.patch, YARN-9060-trunk.004.patch, 
> YARN-9060-trunk.005.patch, YARN-9060-trunk.006.patch, 
> YARN-9060-trunk.007.patch, YARN-9060-trunk.008.patch, 
> YARN-9060-trunk.009.patch, YARN-9060-trunk.010.patch, 
> YARN-9060-trunk.011.patch, YARN-9060-trunk.012.patch, 
> YARN-9060-trunk.013.patch, YARN-9060-trunk.014.patch
>
>
> Due to the cgroups v1 implementation policy in linux kernel, we cannot update 
> the value of the device cgroups controller unless we have the root permission 
> ([here|https://github.com/torvalds/linux/blob/6f0d349d922ba44e4348a17a78ea51b7135965b1/security/device_cgroup.c#L604]).
>  So we need to support this in container-executor for Java layer to invoke.
> This Jira will have three parts:
>  # native c-e module
>  # Java layer code to isolate devices for container (docker and non-docker)
>  # A sample Nvidia GPU plugin



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-8821) GPU hierarchy/topology scheduling support



 [ 
https://issues.apache.org/jira/browse/YARN-8821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhankun Tang updated YARN-8821:
---
Attachment: YARN-8821-trunk.004.patch

> GPU hierarchy/topology scheduling support
> -
>
> Key: YARN-8821
> URL: https://issues.apache.org/jira/browse/YARN-8821
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
>Priority: Major
> Attachments: YARN-8821-trunk.001.patch, YARN-8821-trunk.002.patch, 
> YARN-8821-trunk.003.patch, YARN-8821-trunk.004.patch
>
>
> GPU topology affects performance dramatically. There's been a discussion in 
> YARN-7481. But we'd like to move related discussions here.
> Please note that YARN-8851 will provide a pluggable device framework which 
> can support plugin custom scheduler. And based on the framework, GPU plugin 
> could have own topology scheduler. The proposed patch has a topology 
> algorithm implemented as below:
>  *Step 1*. When allocating devices, parse the output of "nvidia-smi topo -m" 
> to build a hash map whose key is all pairs of GPUs and the value is the 
> communication cost between the two. The map is like \{"0 - 1"=> 2, "0 - 
> 2"=>4, ...} which means the minimum cost of GPU 0 to 1 is 2. The cost is set 
> based on the connection type.
> *Step 2*. And then it constructs a _+cost table+_ which caches all 
> combinations of GPUs and corresponding cost between them and cache it. The 
> cost table is a map whose structure is like
>  
> {code:java}
> { 2=>{[0,1]=>2,..},
>   3=>{[0,1,2]=>10,..},
>   4=>{[0,1,2,3]=>18}}.
> {code}
> The key of the map is the count of GPUs, the value of it is a map whose key 
> is the combination of GPUs and the value is the calculated communication cost 
> of the numbers of GPUs. The cost calculation algorithm is to sum all 
> non-duplicate pairs of GPU's cost. For instance, the total cost of [0,1,2] 
> GPUs are the sum of cost "0 - 1", "0 - 2" and "1 - 2". And each cost can get 
> from the map built in step 1.
> *Step 3*. After the cost table is built, when allocating GPUs based on 
> topology, we provide two policy which container can set through an 
> environment variable "NVIDIA_TOPO_POLICY". The value can be either "PACK" or 
> "SPREAD". The "PACK" means it prefers faster GPU-GPU communication. The 
> "SPREAD" means it prefers faster CPU-GPU communication( since GPUs are not 
> using the same bus to CPU). And the key difference of the two policy is the 
> sort order of the inner map in the cost table. For instance, let's assume 2 
> GPUs is wanted. The costTable.get(2) would return a map containing all 
> combinations of two GPUs and their cost. If the policy is "PACK", we'll sort 
> the map by cost in ascending order. The first entry will be the GPUs has 
> minimum GPU-GPU cost. If the policy is "SPREAD", we sort it in descending 
> order and get the first one which is the highest GPU-GPU cost which means 
> lowest CPU-GPU costs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9208) Distributed shell allow LocalResourceVisibility to be specified

2019-01-30 Thread Bibin A Chundatt (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756890#comment-16756890
 ] 

Bibin A Chundatt commented on YARN-9208:


[~Prabhu Joseph] Sorry for the delay.

 

Few comments
{code:java}
764 case "true": return LocalResourceVisibility.PUBLIC;
765 case "false": return LocalResourceVisibility.PRIVATE;
{code}
Better to give visibility type as argument (public,private,application) for 
files
{code:java}
744 // Default Visibilty is set to APPLICATION type
745 int size = fileNames.size();
746 if (size == visibilities.size()) {
747   for (int i=0; ivisibilityType,filename2visibilityType" . Rename key 
(localized_files) if required. Limitation of "size == visibilities.size()" we 
could avoid.
{quote}The Client determines the visibilities (PRIVATE, PUBLIC and default 
APPLICATION) based upon the permission of the input file. Also supports input 
files from Hdfs.
{quote}
Does file scheme allow PUBLIC definition ??

> Distributed shell allow LocalResourceVisibility to be specified
> ---
>
> Key: YARN-9208
> URL: https://issues.apache.org/jira/browse/YARN-9208
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Bibin A Chundatt
>Assignee: Prabhu Joseph
>Priority: Minor
> Attachments: YARN-9208-001.patch
>
>
> YARN-9008 add feature to add list of files to be localized.
> Would be great to have Visibility type too. Allows testing of PRIVATE and 
> PUBLIC type too



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9060) [YARN-8851] Phase 1 - Support device isolation and use the Nvidia GPU plugin as an example



 [ 
https://issues.apache.org/jira/browse/YARN-9060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhankun Tang updated YARN-9060:
---
Attachment: YARN-9060-trunk.014.patch

> [YARN-8851] Phase 1 - Support device isolation and use the Nvidia GPU plugin 
> as an example
> --
>
> Key: YARN-9060
> URL: https://issues.apache.org/jira/browse/YARN-9060
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
>Priority: Major
> Attachments: YARN-9060-trunk.001.patch, YARN-9060-trunk.002.patch, 
> YARN-9060-trunk.003.patch, YARN-9060-trunk.004.patch, 
> YARN-9060-trunk.005.patch, YARN-9060-trunk.006.patch, 
> YARN-9060-trunk.007.patch, YARN-9060-trunk.008.patch, 
> YARN-9060-trunk.009.patch, YARN-9060-trunk.010.patch, 
> YARN-9060-trunk.011.patch, YARN-9060-trunk.012.patch, 
> YARN-9060-trunk.013.patch, YARN-9060-trunk.014.patch
>
>
> Due to the cgroups v1 implementation policy in linux kernel, we cannot update 
> the value of the device cgroups controller unless we have the root permission 
> ([here|https://github.com/torvalds/linux/blob/6f0d349d922ba44e4348a17a78ea51b7135965b1/security/device_cgroup.c#L604]).
>  So we need to support this in container-executor for Java layer to invoke.
> This Jira will have three parts:
>  # native c-e module
>  # Java layer code to isolate devices for container (docker and non-docker)
>  # A sample Nvidia GPU plugin



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9256) Make ATSv2 compilation default with hbase.profile=2.0



 [ 
https://issues.apache.org/jira/browse/YARN-9256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-9256:

Attachment: YARN-9256.01.patch

> Make ATSv2 compilation default with hbase.profile=2.0
> -
>
> Key: YARN-9256
> URL: https://issues.apache.org/jira/browse/YARN-9256
> Project: Hadoop YARN
>  Issue Type: Task
>Reporter: Rohith Sharma K S
>Priority: Major
> Attachments: YARN-9256.01.patch
>
>
> By default Hadoop compiles with hbase.profile one which corresponds to 
> hbase.version=1.4 for ATSv2. Change compilation to hbase.profile=2.0 by 
> default in trunk. 
> This JIRA is to discuss for any concerns. 
> cc:/ [~vrushalic]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9256) Make ATSv2 compilation default with hbase.profile=2.0



 [ 
https://issues.apache.org/jira/browse/YARN-9256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-9256:

Target Version/s: 3.3.0

> Make ATSv2 compilation default with hbase.profile=2.0
> -
>
> Key: YARN-9256
> URL: https://issues.apache.org/jira/browse/YARN-9256
> Project: Hadoop YARN
>  Issue Type: Task
>Reporter: Rohith Sharma K S
>Priority: Major
>
> By default Hadoop compiles with hbase.profile one which corresponds to 
> hbase.version=1.4 for ATSv2. Change compilation to hbase.profile=2.0 by 
> default in trunk. 
> This JIRA is to discuss for any concerns. 
> cc:/ [~vrushalic]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-9256) Make ATSv2 compilation default with hbase.profile=2.0

Rohith Sharma K S created YARN-9256:
---

 Summary: Make ATSv2 compilation default with hbase.profile=2.0
 Key: YARN-9256
 URL: https://issues.apache.org/jira/browse/YARN-9256
 Project: Hadoop YARN
  Issue Type: Task
Reporter: Rohith Sharma K S


By default Hadoop compiles with hbase.profile one which corresponds to 
hbase.version=1.4 for ATSv2. Change compilation to hbase.profile=2.0 by default 
in trunk. 

This JIRA is to discuss for any concerns. 
cc:/ [~vrushalic]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9099) GpuResourceAllocator#getReleasingGpus calculates number of GPUs in a wrong way

2019-01-30 Thread Hudson (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756845#comment-16756845
 ] 

Hudson commented on YARN-9099:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15859 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/15859/])
YARN-9099. GpuResourceAllocator#getReleasingGpus calculates number of (sunilg: 
rev 71c49fa60faad2504b0411979a6e46e595b97a85)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/gpu/GpuResourceAllocator.java


> GpuResourceAllocator#getReleasingGpus calculates number of GPUs in a wrong way
> --
>
> Key: YARN-9099
> URL: https://issues.apache.org/jira/browse/YARN-9099
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: YARN-9099.001.patch, YARN-9099.002.patch
>
>
> getReleasingGpus plays an important role in the calculation which happens 
> when GpuAllocator assign GPUs to a container, see: 
> GpuResourceAllocator#internalAssignGpus.
> If multiple GPUs are assigned to the same container, getReleasingGpus will 
> return an invalid number.
> The iterator goes over on mappings of (GPU device, container ID) and it 
> retrieves the container by its ID the number of times the container ID is 
> mapped to any device.
> Then for every container, the resource value for the GPU resource is added to 
> a running sum.
> Obviously, if a container is mapped to 2 or more devices, then the 
> container's GPU resource counter is added to the running sum as many times as 
> the number of GPU devices the container has.
> Example: 
> Let's suppose {{usedDevices}} contains these mappings: 
> - (GPU1, container1)
> - (GPU2, container1)
> - (GPU3, container2)
> GPU resource value is 2 for container1 and 
> GPU resource value is 1 for container2.
> Then, if container1 is in a running state, getReleasingGpus will return 4 
> instead of 2.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9161) Absolute resources of capacity scheduler doesn't support GPU and FPGA



[ 
https://issues.apache.org/jira/browse/YARN-9161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756839#comment-16756839
 ] 

Sunil Govindan commented on YARN-9161:
--

Checking jenkins again to rule out the test failure.

> Absolute resources of capacity scheduler doesn't support GPU and FPGA
> -
>
> Key: YARN-9161
> URL: https://issues.apache.org/jira/browse/YARN-9161
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler
>Reporter: Zac Zhou
>Assignee: Zac Zhou
>Priority: Major
> Attachments: YARN-9161.001.patch, YARN-9161.002.patch, 
> YARN-9161.003.patch, YARN-9161.004.patch, YARN-9161.005.patch, 
> YARN-9161.006.patch, YARN-9161.007.patch, YARN-9161.008.patch, 
> YARN-9161.009.patch
>
>
> As the enum CapacitySchedulerConfiguration.AbsoluteResourceType only has two 
> elements: memory and vcores, which would filter out absolute resources 
> configuration of gpu and fpga in 
> AbstractCSQueue.updateConfigurableResourceRequirement. 
> This issue would cause gpu and fpga can't be allocated correctly



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9251) Build failure for -Dhbase.profile=2.0



[ 
https://issues.apache.org/jira/browse/YARN-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756840#comment-16756840
 ] 

Rohith Sharma K S commented on YARN-9251:
-

Thanks [~ajisakaa] for committing patch.

> Build failure for -Dhbase.profile=2.0
> -
>
> Key: YARN-9251
> URL: https://issues.apache.org/jira/browse/YARN-9251
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Blocker
> Fix For: 3.3.0
>
> Attachments: HADOOP-16088.01.patch
>
>
> Post HADOOP-14178, hadoop build failure due to incorrect pom.xml. 
> {noformat}
> HW12723:hadoop rsharmaks$ mvn clean install -DskipTests -DskipShade 
> -Dhbase.profile=2.0
> [INFO] Scanning for projects...
> [ERROR] [ERROR] Some problems were encountered while processing the POMs:
> [ERROR] 'dependencies.dependency.version' for org.mockito:mockito-all:jar is 
> missing. @ line 485, column 21
>  @
> [ERROR] The build could not read 1 project -> [Help 1]
> [ERROR]
> [ERROR]   The project 
> org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-tests:3.3.0-SNAPSHOT
>  
> (/Users/rsharmaks/Repos/Apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/pom.xml)
>  has 1 error
> [ERROR] 'dependencies.dependency.version' for org.mockito:mockito-all:jar 
> is missing. @ line 485, column 21
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please 
> read the following articles:
> [ERROR] [Help 1] 
> http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
> {noformat}
> cc:/ [~ajisakaa]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9099) GpuResourceAllocator#getReleasingGpus calculates number of GPUs in a wrong way



 [ 
https://issues.apache.org/jira/browse/YARN-9099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sunil Govindan updated YARN-9099:
-
Summary: GpuResourceAllocator#getReleasingGpus calculates number of GPUs in 
a wrong way  (was: GpuResourceAllocator.getReleasingGpus calculates number of 
GPUs in a wrong way)

> GpuResourceAllocator#getReleasingGpus calculates number of GPUs in a wrong way
> --
>
> Key: YARN-9099
> URL: https://issues.apache.org/jira/browse/YARN-9099
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: YARN-9099.001.patch, YARN-9099.002.patch
>
>
> getReleasingGpus plays an important role in the calculation which happens 
> when GpuAllocator assign GPUs to a container, see: 
> GpuResourceAllocator#internalAssignGpus.
> If multiple GPUs are assigned to the same container, getReleasingGpus will 
> return an invalid number.
> The iterator goes over on mappings of (GPU device, container ID) and it 
> retrieves the container by its ID the number of times the container ID is 
> mapped to any device.
> Then for every container, the resource value for the GPU resource is added to 
> a running sum.
> Obviously, if a container is mapped to 2 or more devices, then the 
> container's GPU resource counter is added to the running sum as many times as 
> the number of GPU devices the container has.
> Example: 
> Let's suppose {{usedDevices}} contains these mappings: 
> - (GPU1, container1)
> - (GPU2, container1)
> - (GPU3, container2)
> GPU resource value is 2 for container1 and 
> GPU resource value is 1 for container2.
> Then, if container1 is in a running state, getReleasingGpus will return 4 
> instead of 2.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9191) Add cli option in DS to support enforceExecutionType in resource requests.

2019-01-30 Thread Abhishek Modi (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756820#comment-16756820
 ] 

Abhishek Modi commented on YARN-9191:
-

[~elgoiri] all the test passes locally for me. 

> Add cli option in DS to support enforceExecutionType in resource requests.
> --
>
> Key: YARN-9191
> URL: https://issues.apache.org/jira/browse/YARN-9191
> Project: Hadoop YARN
>  Issue Type: Task
>Reporter: Abhishek Modi
>Assignee: Abhishek Modi
>Priority: Major
> Attachments: YARN-9191.001.patch, YARN-9191.002.patch, 
> YARN-9191.003.patch, YARN-9191.004.patch, YARN-9191.005.patch, 
> YARN-9191.006.patch
>
>
> This JIRA proposes to expose cli option to allow users to additionally 
> specify the value enforceExecutionType flag (introduced in YARN-5180).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9248) RMContainerImpl:Invalid event: ACQUIRED at KILLED



[ 
https://issues.apache.org/jira/browse/YARN-9248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756809#comment-16756809
 ] 

lujie commented on YARN-9248:
-

i have found the reason why 
TestRMAppAttemptTransitions#testContainerRemovedBeforeAllocate fails, this test 
was introduced in YARN-9194 , and works well previously. Seems some code 
changes in RMAppAttemptImpl$AMContainerAllocatedTransition.transition. 

In patch [^YARN-9248_5.patch], I have fixed the problem, and it works well,(see 
the lateset HADOOP AQ) 

Hope for  review

 

> RMContainerImpl:Invalid event: ACQUIRED at KILLED
> -
>
> Key: YARN-9248
> URL: https://issues.apache.org/jira/browse/YARN-9248
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: lujie
>Assignee: lujie
>Priority: Major
> Attachments: YARN-9248_1.patch, YARN-9248_2.patch, YARN-9248_3.patch, 
> YARN-9248_4.patch, YARN-9248_5.patch
>
>
> {code:java}
> 2019-01-29 11:46:53,596 ERROR 
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: 
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event: 
> ACQUIRED at KILLED
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:475)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:67)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.handleNewContainers(OpportunisticContainerAllocatorAMService.java:351)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.access$100(OpportunisticContainerAllocatorAMService.java:94)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService$OpportunisticAMSProcessor.allocate(OpportunisticContainerAllocatorAMService.java:197)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:424)
> at 
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at 
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:530)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:878)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2830)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9252) Allocation Tag Namespace support in Distributed Shell



[ 
https://issues.apache.org/jira/browse/YARN-9252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756796#comment-16756796
 ] 

Weiwei Yang commented on YARN-9252:
---

Moving to PC enhancement umbrella.

> Allocation Tag Namespace support in Distributed Shell
> -
>
> Key: YARN-9252
> URL: https://issues.apache.org/jira/browse/YARN-9252
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: distributed-shell
>Affects Versions: 3.1.1
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
>
> Distributed Shell supports placement constraint but allocation Tag Namespace 
> is not honored.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9253) Add UT to verify Placement Constraint in Distributed Shell



 [ 
https://issues.apache.org/jira/browse/YARN-9253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang updated YARN-9253:
--
Issue Type: Sub-task  (was: Test)
Parent: YARN-8731

> Add UT to verify Placement Constraint in Distributed Shell
> --
>
> Key: YARN-9253
> URL: https://issues.apache.org/jira/browse/YARN-9253
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Affects Versions: 3.1.1
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
>
> Add UT to verify Placement Constraint in Distributed Shell added as part of 
> YARN-7745



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9252) Allocation Tag Namespace support in Distributed Shell



 [ 
https://issues.apache.org/jira/browse/YARN-9252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang updated YARN-9252:
--
Issue Type: Sub-task  (was: Bug)
Parent: YARN-8731

> Allocation Tag Namespace support in Distributed Shell
> -
>
> Key: YARN-9252
> URL: https://issues.apache.org/jira/browse/YARN-9252
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: distributed-shell
>Affects Versions: 3.1.1
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
>
> Distributed Shell supports placement constraint but allocation Tag Namespace 
> is not honored.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9250) hadoop-yarn-server-nodemanager build failed: make failed with error code 2



 [ 
https://issues.apache.org/jira/browse/YARN-9250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

charlie mao updated YARN-9250:
--
Description: 
when i compile hadoop-3.2.0 release,i encountered the following errors：

[ERROR] Failed to execute goal 
org.apache.hadoop:hadoop-maven-plugins:3.2.0:cmake-compile (cmake-compile) on 
project hadoop-yarn-server-nodemanager: make failed with error code 2 -> [Help 
1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal 
org.apache.hadoop:hadoop-maven-plugins:3.2.0:cmake-compile (cmake-compile) on 
project hadoop-yarn-server-nodemanager: make failed with error code 2
 at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
 at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
 at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
 at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
 at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
 at 
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
 at 
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
 at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
 at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
 at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
 at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
 at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
 at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at 
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
 at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
 at 
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
 at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: make failed with 
error code 2
 at 
org.apache.hadoop.maven.plugin.cmakebuilder.CompileMojo.runMake(CompileMojo.java:231)
 at 
org.apache.hadoop.maven.plugin.cmakebuilder.CompileMojo.execute(CompileMojo.java:98)
 at 
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
 at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
 ... 20 more
[ERROR]
[ERROR]
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn  -rf :hadoop-yarn-server-nodemanager

 

my compiling environments：

jdk .8.0_181.8.0_181

maven:3.3.9(3.6.0)

cmake version 3.12.0



  was:
when i compile hadoop-3.2.0 release,i encounter following errors：

[ERROR] Failed to execute goal 
org.apache.hadoop:hadoop-maven-plugins:3.2.0:cmake-compile (cmake-compile) on 
project hadoop-yarn-server-nodemanager: make failed with error code 2 -> [Help 
1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal 
org.apache.hadoop:hadoop-maven-plugins:3.2.0:cmake-compile (cmake-compile) on 
project hadoop-yarn-server-nodemanager: make failed with error code 2
 at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
 at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
 at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
 at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
 at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
 at 
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
 at 
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
 at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
 at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
 at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
 at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
 at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
 at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at

[jira] [Created] (YARN-9254) Externalize Solr data storage

2019-01-30 Thread Eric Yang (JIRA)

Eric Yang created YARN-9254:
---

 Summary: Externalize Solr data storage
 Key: YARN-9254
 URL: https://issues.apache.org/jira/browse/YARN-9254
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Eric Yang


Application catalog contains embedded Solr.  By default, Solr data is stored in 
temp space of the docker container.  For user who likes to persist Solr data on 
HDFS, it would be nice to have a way to pass solr.hdfs.home setting to embedded 
Solr to externalize Solr data storage.  This also implies passing Kerberos 
credential settings to Solr JVM in order to access secure HDFS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9118) Handle issues with parsing user defined GPU devices in GpuDiscoverer



[ 
https://issues.apache.org/jira/browse/YARN-9118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756618#comment-16756618
 ] 

Hadoop QA commented on YARN-9118:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m  9s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
25s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
6s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 24s{color} | {color:orange} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
 The patch generated 1 new + 10 unchanged - 5 fixed = 11 total (was 15) {color} 
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
15m 35s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 20m 28s{color} 
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
25s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 80m 40s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.nodemanager.nodelabels.TestConfigurationNodeAttributesProvider
 |
|   | 
hadoop.yarn.server.nodemanager.containermanager.resourceplugin.gpu.TestGpuDiscoverer
 |
|   | 
hadoop.yarn.server.nodemanager.containermanager.linux.resources.gpu.TestGpuResourceHandler
 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9118 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12956955/YARN-9118.005.patch |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 18c7276eaaa4 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct 
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c354195 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |

[jira] [Created] (YARN-9255) Improve recommend applications order

2019-01-30 Thread Eric Yang (JIRA)

Eric Yang created YARN-9255:
---

 Summary: Improve recommend applications order
 Key: YARN-9255
 URL: https://issues.apache.org/jira/browse/YARN-9255
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Eric Yang


When there is no search term in application catalog, the recommended 
application list is random.  The relevance can be fine tuned to be sorted by 
number of downloads or alphabetic order.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7129) Application Catalog for YARN applications

2019-01-30 Thread Eric Yang (JIRA)

[
https://issues.apache.org/jira/browse/YARN-7129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756585#comment-16756585
]

Eric Yang commented on YARN-7129:
-

[~billie.rinaldi] Thank you for the review.

{quote}* Service catalog might be a better name, since the catalog handles YARN
services and not arbitrary applications. In that case I’d suggest moving the
module to
hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-catalog.{quote}

Hadoop-yarn-services is the default implementation of YARN Application Master.
I view application catalog as independent application that depends on the
Hadoop-yarn-services. Hence, I think it is more modular to keep application
catalog as a peer application from hadoop-yarn-services to prevent overloading
hadoop-yarn-services project.

{quote}* NOTICE says “binary distribution of this product bundles binaries of
___” but it should be “source distribution bundles ___.” Also, this information
should go in LICENSE instead of NOTICE for non-ASLv2 permissive licenses (see
http://www.apache.org/dev/licensing-howto.html#permissive-deps). It would be
helpful to include the path(s) to the dependencies, so it’s easier to tell when
included source code has proper handling in LICENSE and NOTICE and when it
doesn’t. I have not checked whether all the external code added in this patch
has been included properly in L yet.{quote}

Most of the nodejs projects are sourced to run unit tests. Similar to maven
plugins, no NOTICE or LICENSE are included for the build tools. jQuery, and
AngularJS are not part of the source distribution, but integrated into binary
web application. This is the reason that the NOTICE file includes binary
distribution of this product bundles binaries of jQuery and AngularJS.

{quote}* There are files named bootstrap-hwx.js and bootstrap-hwx.min.js.{quote}

Will remove hwx from filename in the next patch.

{quote}* 8080 is a difficult port. If you ran the catalog on an
Ambari-installed cluster, it would fail if it came up on the Ambari server
host. Does the app catalog need to have net=host? If not, the port wouldn’t be
a problem.{quote}

App catalog does not require net=host. In Hadoop 3.3,
YARN_CONTAINER_RUNTIME_DOCKER_PORTS_MAPPING flag can be used to expose app
catalog on ephemeral port.

{quote}* I don’t think we should set net=host for the examples, because that
setting is not recommended for most containers. Also, for the services using
8080, they could run into conflicts when net=host.{quote}

Will change to use YARN_CONTAINER_RUNTIME_DOCKER_PORTS flag in the next patch.

{quote}* I think it would be better to have fewer examples in samples.xml and
make sure that they all work (the pcjs image doesn’t exist on dockerhub, and we
might not want to seem to be “endorsing” all the example images that are
currently included). Maybe just include httpd and Jenkins? Registry would also
be a reasonable candidate, but it didn’t work when I tried it with insecure
limit users mode (it failed to make a dir when pushing an image to it; maybe
would work as a privileged container with image name
library/registry:latest).{quote}

I will keep Jenkins, httpd, and Docker Registry as example applications. For
private cloud, it is essential to have private Docker Registry, and this
appears to be the best way to deploy with ease.

{quote}* entrypoint.sh needs modifications to make insecure mode possible
(maybe checking for empty KEYTAB variable would be sufficient, since -e returns
true for an empty variable).{quote}

Will include this change in the next patch.

{quote}* Need better defaults for memory requests. When I tested, the catalog
and Jenkins containers were killed due to OOM. 2048 for the catalog and 1024
for Jenkins worked for me.{quote}

Will include this change in the next patch.

{quote}* I had to set JAVA_HOME in the catalog service json because the
container and host didn’t have the same JAVA_HOME.
* It would be good to include the service json needed to launch the catalog.
We’d need to make the catalog Docker image version a variable for maven to fill
in during the build process. Maybe the catalog could be one of the service
examples, like sleeper and httpd.{quote}

Will include service json in example jar file for easy deployment.

{quote}* Downloading from archive.apache.org is discouraged. Is there anything
else we can do instead for the solr download?{quote}

Solr download is problematic because Apache mirror only keeps the most recent
releases. I am open to suggestion other than getting from archive.apache.org.
I don't know other good way to get fixed version of Solr tarball at this time.

{quote}* Applications launched by the catalog are run as the root user
(presumably because the catalog is a privileged container); at least that’s
what is happening in insecure mode. The catalog should have user auth and
launch the apps as

[jira] [Commented] (YARN-9191) Add cli option in DS to support enforceExecutionType in resource requests.

2019-01-30 Thread JIRA



[ 
https://issues.apache.org/jira/browse/YARN-9191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756552#comment-16756552
 ] 

Íñigo Goiri commented on YARN-9191:
---

[^YARN-9191.006.patch] looks good.
However the Yetus run doesn't seem to show the added test.
[~abmodi] any idea on what's the status of the unit test?

> Add cli option in DS to support enforceExecutionType in resource requests.
> --
>
> Key: YARN-9191
> URL: https://issues.apache.org/jira/browse/YARN-9191
> Project: Hadoop YARN
>  Issue Type: Task
>Reporter: Abhishek Modi
>Assignee: Abhishek Modi
>Priority: Major
> Attachments: YARN-9191.001.patch, YARN-9191.002.patch, 
> YARN-9191.003.patch, YARN-9191.004.patch, YARN-9191.005.patch, 
> YARN-9191.006.patch
>
>
> This JIRA proposes to expose cli option to allow users to additionally 
> specify the value enforceExecutionType flag (introduced in YARN-5180).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9213) RM Web UI does not show custom resource allocations for containers page



[ 
https://issues.apache.org/jira/browse/YARN-9213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756547#comment-16756547
 ] 

Szilard Nemeth commented on YARN-9213:
--

Hi [~pbacsko]!

Thanks for the review!

No, this is not only a safety net. If you check the calls of 
\{{getResourceAsString}}, you can see that is invoked with standard resources 
(memory, vcores) as well.

See this part of the patch: 
{code:java}
sb.append(getResourceAsString(ResourceInformation.MEMORY_URI, 
allocatedResources.get(ResourceInformation.MEMORY_URI))).append(", "); 
sb.append(getResourceAsString(ResourceInformation.VCORES_URI, 
allocatedResources.get(ResourceInformation.VCORES_URI)));
{code}




> RM Web UI does not show custom resource allocations for containers page
> ---
>
> Key: YARN-9213
> URL: https://issues.apache.org/jira/browse/YARN-9213
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: YARN-9213.001.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-9118) Handle issues with parsing user defined GPU devices in GpuDiscoverer



[ 
https://issues.apache.org/jira/browse/YARN-9118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756544#comment-16756544
 ] 

Szilard Nemeth edited comment on YARN-9118 at 1/30/19 8:48 PM:
---

Hi [~pbacsko]!
 Thanks for your review comments once again!

When lastDiscoveredGpuInformation.getGpus() is null, the return value of 
{{parseGpuDevicesFromAutoDiscoveredGpuInfo()}} will be an empty-list since no 
GPU devices are found. 
 The error is handled elsewhere, see the 2 callers of 
{{GpuDiscoverer.getInstance().getGpusUsableByYarn()}}.
 Anyway, I added a debug log here as well to indicate that no GPU was found by 
the auto-discovery binary, so this case can obviously happen.

Please note that the latest patch contains some additional cleanup, but very 
minor ones.


was (Author: snemeth):
Hi [~pbacsko]!
Thanks for your review comments once again!

When lastDiscoveredGpuInformation.getGpus() is null, the return value of 
{{parseGpuDevicesFromAutoDiscoveredGpuInfo()}} will be an empty-list since no 
GPU devices are found. 
The error is handled elsewhere, see the 2 callers of 
{{GpuDiscoverer.getInstance().getGpusUsableByYarn()}}.
Anyway, I added a debug log here as well to indicate that no GPU was found.

Please note that the latest patch contains some additional cleanup, but very 
minor ones.

> Handle issues with parsing user defined GPU devices in GpuDiscoverer
> 
>
> Key: YARN-9118
> URL: https://issues.apache.org/jira/browse/YARN-9118
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: YARN-9118.001.patch, YARN-9118.002.patch, 
> YARN-9118.003.patch, YARN-9118.004.patch
>
>
> getGpusUsableByYarn has the following issues: 
> - Duplicate GPU device definitions are not denied: This seems to be the 
> biggest issue as it could increase the number of devices on the node if the 
> device ID is defined 2 or more times.
> - An empty-string is accepted, it works like the user would not want to use 
> auto-discovery and haven't defined any GPU devices: This will result in an 
> empty device list, but the empty-string check is never explicitly there in 
> the code, so this behavior just coincidental.
> - Number validation does not happen on GPU device IDs (separated by commas)
> Many testcases are added as the coverage was already very low.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-9118) Handle issues with parsing user defined GPU devices in GpuDiscoverer



[ 
https://issues.apache.org/jira/browse/YARN-9118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756544#comment-16756544
 ] 

Szilard Nemeth edited comment on YARN-9118 at 1/30/19 8:49 PM:
---

Hi [~pbacsko]!
 Thanks for your review comments once again!

When lastDiscoveredGpuInformation.getGpus() is null, the return value of 
{{parseGpuDevicesFromAutoDiscoveredGpuInfo()}} will be an empty-list since no 
GPU devices are found. 
 The error is handled elsewhere, see the 2 callers of 
{{GpuDiscoverer.getInstance().getGpusUsableByYarn()}}.
 Anyway, I added a debug log here as well to indicate that no GPU was found by 
the auto-discovery binary, so this case can obviously happen.

Please note that the latest patch contains some additional cleanup, but very 
minor ones.

Please check the updated patch! Thanks!


was (Author: snemeth):
Hi [~pbacsko]!
 Thanks for your review comments once again!

When lastDiscoveredGpuInformation.getGpus() is null, the return value of 
{{parseGpuDevicesFromAutoDiscoveredGpuInfo()}} will be an empty-list since no 
GPU devices are found. 
 The error is handled elsewhere, see the 2 callers of 
{{GpuDiscoverer.getInstance().getGpusUsableByYarn()}}.
 Anyway, I added a debug log here as well to indicate that no GPU was found by 
the auto-discovery binary, so this case can obviously happen.

Please note that the latest patch contains some additional cleanup, but very 
minor ones.

> Handle issues with parsing user defined GPU devices in GpuDiscoverer
> 
>
> Key: YARN-9118
> URL: https://issues.apache.org/jira/browse/YARN-9118
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: YARN-9118.001.patch, YARN-9118.002.patch, 
> YARN-9118.003.patch, YARN-9118.004.patch, YARN-9118.005.patch
>
>
> getGpusUsableByYarn has the following issues: 
> - Duplicate GPU device definitions are not denied: This seems to be the 
> biggest issue as it could increase the number of devices on the node if the 
> device ID is defined 2 or more times.
> - An empty-string is accepted, it works like the user would not want to use 
> auto-discovery and haven't defined any GPU devices: This will result in an 
> empty device list, but the empty-string check is never explicitly there in 
> the code, so this behavior just coincidental.
> - Number validation does not happen on GPU device IDs (separated by commas)
> Many testcases are added as the coverage was already very low.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9118) Handle issues with parsing user defined GPU devices in GpuDiscoverer



 [ 
https://issues.apache.org/jira/browse/YARN-9118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szilard Nemeth updated YARN-9118:
-
Attachment: YARN-9118.005.patch

> Handle issues with parsing user defined GPU devices in GpuDiscoverer
> 
>
> Key: YARN-9118
> URL: https://issues.apache.org/jira/browse/YARN-9118
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: YARN-9118.001.patch, YARN-9118.002.patch, 
> YARN-9118.003.patch, YARN-9118.004.patch, YARN-9118.005.patch
>
>
> getGpusUsableByYarn has the following issues: 
> - Duplicate GPU device definitions are not denied: This seems to be the 
> biggest issue as it could increase the number of devices on the node if the 
> device ID is defined 2 or more times.
> - An empty-string is accepted, it works like the user would not want to use 
> auto-discovery and haven't defined any GPU devices: This will result in an 
> empty device list, but the empty-string check is never explicitly there in 
> the code, so this behavior just coincidental.
> - Number validation does not happen on GPU device IDs (separated by commas)
> Many testcases are added as the coverage was already very low.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9118) Handle issues with parsing user defined GPU devices in GpuDiscoverer



[ 
https://issues.apache.org/jira/browse/YARN-9118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756544#comment-16756544
 ] 

Szilard Nemeth commented on YARN-9118:
--

Hi [~pbacsko]!
Thanks for your review comments once again!

When lastDiscoveredGpuInformation.getGpus() is null, the return value of 
{{parseGpuDevicesFromAutoDiscoveredGpuInfo()}} will be an empty-list since no 
GPU devices are found. 
The error is handled elsewhere, see the 2 callers of 
{{GpuDiscoverer.getInstance().getGpusUsableByYarn()}}.
Anyway, I added a debug log here as well to indicate that no GPU was found.

Please note that the latest patch contains some additional cleanup, but very 
minor ones.

> Handle issues with parsing user defined GPU devices in GpuDiscoverer
> 
>
> Key: YARN-9118
> URL: https://issues.apache.org/jira/browse/YARN-9118
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: YARN-9118.001.patch, YARN-9118.002.patch, 
> YARN-9118.003.patch, YARN-9118.004.patch
>
>
> getGpusUsableByYarn has the following issues: 
> - Duplicate GPU device definitions are not denied: This seems to be the 
> biggest issue as it could increase the number of devices on the node if the 
> device ID is defined 2 or more times.
> - An empty-string is accepted, it works like the user would not want to use 
> auto-discovery and haven't defined any GPU devices: This will result in an 
> empty device list, but the empty-string check is never explicitly there in 
> the code, so this behavior just coincidental.
> - Number validation does not happen on GPU device IDs (separated by commas)
> Many testcases are added as the coverage was already very low.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9060) [YARN-8851] Phase 1 - Support device isolation and use the Nvidia GPU plugin as an example



[ 
https://issues.apache.org/jira/browse/YARN-9060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756482#comment-16756482
 ] 

Hadoop QA commented on YARN-9060:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 4 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  5m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
17m 57s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  7m 
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  7m 24s{color} 
| {color:red} hadoop-yarn-project_hadoop-yarn generated 6 new + 133 unchanged - 
0 fixed = 139 total (was 133) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m 15s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 2 new + 5 unchanged - 0 fixed = 7 total (was 5) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  4m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 57s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}138m 19s{color} 
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 20m 46s{color} 
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
42s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}246m 17s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.nodemanager.nodelabels.TestConfigurationNodeAttributesProvider
 |
|   | 
hadoop.yarn.server.resourcemanager.rmapp.attempt.TestRMAppAttemptTransitions |
|   |

[jira] [Commented] (YARN-9191) Add cli option in DS to support enforceExecutionType in resource requests.



[ 
https://issues.apache.org/jira/browse/YARN-9191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756483#comment-16756483
 ] 

Hadoop QA commented on YARN-9191:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
15m 45s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  8m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m 47s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 15s{color} 
| {color:red} hadoop-yarn-applications-distributedshell in the patch failed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
27s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
37s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 96m 38s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9191 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12956925/YARN-9191.006.patch |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 14abfec34791 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct 
5 08:56:16 UTC

[jira] [Created] (YARN-9253) Add UT to verify Placement Constraint in Distributed Shell

Prabhu Joseph created YARN-9253:
---

 Summary: Add UT to verify Placement Constraint in Distributed Shell
 Key: YARN-9253
 URL: https://issues.apache.org/jira/browse/YARN-9253
 Project: Hadoop YARN
  Issue Type: Test
Affects Versions: 3.1.1
Reporter: Prabhu Joseph
Assignee: Prabhu Joseph


Add UT to verify Placement Constraint in Distributed Shell added as part of 
YARN-7745



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Assigned] (YARN-9211) The yarn resourcemanager project keeps failing with " There was a timeout or other error in the fork" error



 [ 
https://issues.apache.org/jira/browse/YARN-9211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prabhu Joseph reassigned YARN-9211:
---

Assignee: Prabhu Joseph

> The yarn resourcemanager project keeps failing with " There was a timeout or 
> other error in the fork" error
> ---
>
> Key: YARN-9211
> URL: https://issues.apache.org/jira/browse/YARN-9211
> Project: Hadoop YARN
>  Issue Type: Test
>  Components: test
>Affects Versions: 3.1.2
>Reporter: Aihua Xu
>Assignee: Prabhu Joseph
>Priority: Major
>
> Recently notice that the build keeps failing in resourcemanager project with 
> " There was a timeout or other error in the fork". 
> Here is the part of the log, but I don't see any UT failures. 
> {noformat}
> [WARNING] Tests run: 2445, Failures: 0, Errors: 0, Skipped: 7
> [INFO] 
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 01:32 h
> [INFO] Finished at: 2019-01-18T11:30:20+00:00
> [INFO] Final Memory: 23M/773M
> [INFO] 
> 
> [WARNING] The requested profile "parallel-tests" could not be activated 
> because it does not exist.
> [WARNING] The requested profile "native" could not be activated because it 
> does not exist.
> [WARNING] The requested profile "yarn-ui" could not be activated because it 
> does not exist.
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M1:test (default-test) 
> on project hadoop-yarn-server-resourcemanager: There was a timeout or other 
> error in the fork -> [Help 1]
> {noformat}
> Here is a job https://builds.apache.org/job/PreCommit-YARN-Build/23107/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9248) RMContainerImpl:Invalid event: ACQUIRED at KILLED



[ 
https://issues.apache.org/jira/browse/YARN-9248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756397#comment-16756397
 ] 

Hadoop QA commented on YARN-9248:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m 40s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
34s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m 59s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 93m 22s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
27s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}154m 17s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9248 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12956913/YARN-9248_5.patch |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  |
| uname | Linux bc00c6476a84 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct 
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / a3a9ae3 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit | 
https://builds.apache.org/job/PreCommit-YARN-Build/23229/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/23229/testReport/ |
| Max. process+thread count | 956 (vs. ulimit of 1) |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 |
| Console output |

[jira] [Updated] (YARN-9191) Add cli option in DS to support enforceExecutionType in resource requests.

2019-01-30 Thread Abhishek Modi (JIRA)



 [ 
https://issues.apache.org/jira/browse/YARN-9191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Modi updated YARN-9191:

Attachment: YARN-9191.006.patch

> Add cli option in DS to support enforceExecutionType in resource requests.
> --
>
> Key: YARN-9191
> URL: https://issues.apache.org/jira/browse/YARN-9191
> Project: Hadoop YARN
>  Issue Type: Task
>Reporter: Abhishek Modi
>Assignee: Abhishek Modi
>Priority: Major
> Attachments: YARN-9191.001.patch, YARN-9191.002.patch, 
> YARN-9191.003.patch, YARN-9191.004.patch, YARN-9191.005.patch, 
> YARN-9191.006.patch
>
>
> This JIRA proposes to expose cli option to allow users to additionally 
> specify the value enforceExecutionType flag (introduced in YARN-5180).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7129) Application Catalog for YARN applications

2019-01-30 Thread Billie Rinaldi (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-7129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756339#comment-16756339
 ] 

Billie Rinaldi commented on YARN-7129:
--

Thanks for working on this, [~eyang]. The catalog looks like a promising 
feature. I have started reviewing patch 18. Here is a first round of comments:
 - Service catalog might be a better name, since the catalog handles YARN 
services and not arbitrary applications. In that case I’d suggest moving the 
module to 
hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-catalog.
 - NOTICE says “binary distribution of this product bundles binaries of ___” 
but it should be “source distribution bundles ___.” Also, this information 
should go in LICENSE instead of NOTICE for non-ASLv2 permissive licenses (see 
[http://www.apache.org/dev/licensing-howto.html#permissive-deps]). It would be 
helpful to include the path(s) to the dependencies, so it’s easier to tell when 
included source code has proper handling in LICENSE and NOTICE and when it 
doesn’t. I have not checked whether all the external code added in this patch 
has been included properly in L yet.
 - There are files named bootstrap-hwx.js and bootstrap-hwx.min.js.
 - 8080 is a difficult port. If you ran the catalog on an Ambari-installed 
cluster, it would fail if it came up on the Ambari server host. Does the app 
catalog need to have net=host? If not, the port wouldn’t be a problem.
 - I don’t think we should set net=host for the examples, because that setting 
is not recommended for most containers. Also, for the services using 8080, they 
could run into conflicts when net=host.
 - I think it would be better to have fewer examples in samples.xml and make 
sure that they all work (the pcjs image doesn’t exist on dockerhub, and we 
might not want to seem to be “endorsing” all the example images that are 
currently included). Maybe just include httpd and Jenkins? Registry would also 
be a reasonable candidate, but it didn’t work when I tried it with insecure 
limit users mode (it failed to make a dir when pushing an image to it; maybe 
would work as a privileged container with image name library/registry:latest).
 - entrypoint.sh needs modifications to make insecure mode possible (maybe 
checking for empty KEYTAB variable would be sufficient, since -e returns true 
for an empty variable).
 - Need better defaults for memory requests. When I tested, the catalog and 
Jenkins containers were killed due to OOM. 2048 for the catalog and 1024 for 
Jenkins worked for me.
 - I had to set JAVA_HOME in the catalog service json because the container and 
host didn’t have the same JAVA_HOME.
 - It would be good to include the service json needed to launch the catalog. 
We’d need to make the catalog Docker image version a variable for maven to fill 
in during the build process. Maybe the catalog could be one of the service 
examples, like sleeper and httpd.
 - Downloading from archive.apache.org is discouraged. Is there anything else 
we can do instead for the solr download?
 - Applications launched by the catalog are run as the root user (presumably 
because the catalog is a privileged container); at least that’s what is 
happening in insecure mode. The catalog should have user auth and launch the 
apps as the end user. I see there are already tickets open to address this 
issue.
 - We need to work out persistent storage for some of these services (including 
the catalog), or users will get a bad surprise when services or individual 
containers are restarted.
 - Restart doesn’t seem to work. Looks like it is issuing a start, so maybe 
should be named start instead. Restart implies stopping + starting.
 - Could we use AppAdminClient/ApiServiceClient instead of copying the 
getApiClient methods from ApiServiceClient to make the REST calls?
 - After deploying a service, it drops to the bottom of the list on the main 
catalog page. Is that intentional?
 - If you click on a UI link too early, it brings up a URL such as 
{{jenkins.${service_name}.${user}.${domain}:8080}}. It would be better to 
disallow clicking on the link until the variables are populated, if possible.

> Application Catalog for YARN applications
> -
>
> Key: YARN-7129
> URL: https://issues.apache.org/jira/browse/YARN-7129
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: applications
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN Appstore.pdf, YARN-7129.001.patch, 
> YARN-7129.002.patch, YARN-7129.003.patch, YARN-7129.004.patch, 
> YARN-7129.005.patch, YARN-7129.006.patch, YARN-7129.007.patch, 
> YARN-7129.008.patch, YARN-7129.009.patch, YARN-7129.010.patch, 
> YARN-7129.011.patch, YARN-7129.012.patch, YARN-7129.013.patch, 
> YARN-7129.014.patch, YARN-7129.015.patch,

[jira] [Updated] (YARN-1115) Provide optional means for a scheduler to check real user ACLs

2019-01-30 Thread Eric Payne (JIRA)



 [ 
https://issues.apache.org/jira/browse/YARN-1115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Payne updated YARN-1115:
-
Affects Version/s: (was: 0.23.9)
   (was: 2.1.0-beta)
   2.8.5

> Provide optional means for a scheduler to check real user ACLs
> --
>
> Key: YARN-1115
> URL: https://issues.apache.org/jira/browse/YARN-1115
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: capacity scheduler, scheduler
>Affects Versions: 2.8.5
>Reporter: Eric Payne
>Priority: Major
>
> In the framework for secure implementation using UserGroupInformation.doAs 
> (http://hadoop.apache.org/docs/stable/Secure_Impersonation.html), a trusted 
> superuser can submit jobs on behalf of another user in a secure way. In this 
> framework, the superuser is referred to as the real user and the proxied user 
> is referred to as the effective user.
> Currently when a job is submitted as an effective user, the ACLs for the 
> effective user are checked against the queue on which the job is to be run. 
> Depending on an optional configuration, the scheduler should also check the 
> ACLs of the real user if the configuration to do so is set.
> For example, suppose my superuser name is super, and super is configured to 
> securely proxy as joe. Also suppose there is a Hadoop queue named ops which 
> only allows ACLs for super, not for joe.
> When super proxies to joe in order to submit a job to the ops queue, it will 
> fail because joe, as the effective user, does not have ACLs on the ops queue.
> In many cases this is what you want, in order to protect queues that joe 
> should not be using.
> However, there are times when super may need to proxy to many users, and the 
> client running as super just wants to use the ops queue because the ops queue 
> is already dedicated to the client's purpose, and, to keep the ops queue 
> dedicated to that purpose, super doesn't want to open up ACLs to joe in 
> general on the ops queue. Without this functionality, in this case, the 
> client running as super needs to figure out which queue each user has ACLs 
> opened up for, and then coordinate with other tasks using those queues.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-1115) Provide optional means for a scheduler to check real user ACLs

2019-01-30 Thread Eric Payne (JIRA)



 [ 
https://issues.apache.org/jira/browse/YARN-1115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Payne updated YARN-1115:
-
Component/s: capacity scheduler

> Provide optional means for a scheduler to check real user ACLs
> --
>
> Key: YARN-1115
> URL: https://issues.apache.org/jira/browse/YARN-1115
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: capacity scheduler, scheduler
>Affects Versions: 2.1.0-beta, 0.23.9
>Reporter: Eric Payne
>Priority: Major
>
> In the framework for secure implementation using UserGroupInformation.doAs 
> (http://hadoop.apache.org/docs/stable/Secure_Impersonation.html), a trusted 
> superuser can submit jobs on behalf of another user in a secure way. In this 
> framework, the superuser is referred to as the real user and the proxied user 
> is referred to as the effective user.
> Currently when a job is submitted as an effective user, the ACLs for the 
> effective user are checked against the queue on which the job is to be run. 
> Depending on an optional configuration, the scheduler should also check the 
> ACLs of the real user if the configuration to do so is set.
> For example, suppose my superuser name is super, and super is configured to 
> securely proxy as joe. Also suppose there is a Hadoop queue named ops which 
> only allows ACLs for super, not for joe.
> When super proxies to joe in order to submit a job to the ops queue, it will 
> fail because joe, as the effective user, does not have ACLs on the ops queue.
> In many cases this is what you want, in order to protect queues that joe 
> should not be using.
> However, there are times when super may need to proxy to many users, and the 
> client running as super just wants to use the ops queue because the ops queue 
> is already dedicated to the client's purpose, and, to keep the ops queue 
> dedicated to that purpose, super doesn't want to open up ACLs to joe in 
> general on the ops queue. Without this functionality, in this case, the 
> client running as super needs to figure out which queue each user has ACLs 
> opened up for, and then coordinate with other tasks using those queues.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-9252) Allocation Tag Namespace support in Distributed Shell

Prabhu Joseph created YARN-9252:
---

 Summary: Allocation Tag Namespace support in Distributed Shell
 Key: YARN-9252
 URL: https://issues.apache.org/jira/browse/YARN-9252
 Project: Hadoop YARN
  Issue Type: Bug
  Components: distributed-shell
Affects Versions: 3.1.1
Reporter: Prabhu Joseph
Assignee: Prabhu Joseph


Distributed Shell supports placement constraint but allocation Tag Namespace is 
not honored.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-9240) YARN UI 2 footer shows the datetime in different timezone

2019-01-30 Thread Masahiro Tanaka (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756239#comment-16756239
 ] 

Masahiro Tanaka edited comment on YARN-9240 at 1/30/19 3:48 PM:


Hi [~akhilpb],

> So YARN-8747 patch fixes the this issue, right?

Yes. I built Hadoop from the latest source code which includes YARN-8747, and 
confirmed the problem was fixed.

According to [https://github.com/moment/moment-timezone/releases/tag/0.5.1] ,

"moment/moment-timezone" v0.5.1 includes the PR I mentioned 
[above|https://github.com/moment/moment-timezone/pull/302].


was (Author: masatana):
Hi Akhil PB

> So YARN-8747 patch fixes the this issue, right?

Yes. I built Hadoop from the latest source code which includes YARN-8747, and 
confirmed the problem was fixed.

According to [https://github.com/moment/moment-timezone/releases/tag/0.5.1] ,

"moment/moment-timezone" v0.5.1 includes the PR I mentioned 
[above|https://github.com/moment/moment-timezone/pull/302].

> YARN UI 2 footer shows the datetime in different timezone
> -
>
> Key: YARN-9240
> URL: https://issues.apache.org/jira/browse/YARN-9240
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn-ui-v2
>Affects Versions: 3.2.0
> Environment: Windows 10/Firefox 64
> {code}
>   (new Date()).toTimeString()
>   => "09:11:16 GMT+0900 (日本標準時)"
> {code}
>Reporter: Masahiro Tanaka
>Assignee: Masahiro Tanaka
>Priority: Minor
>
> YARN Web UI 2 footer shows the started time about "2019-01-25 16:39" even if 
> the ResourceManager started at "2019-01-26 00:39:34"(UTC), and my PC's 
> localtime is JST.(+09:00 GMT)
> ResourceManager log is below: (Sever time is set as UTC). 
> {code}
> 2019-01-26 00:39:34,619 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: STARTUP_MSG:
> /
> STARTUP_MSG: Starting ResourceManager
> STARTUP_MSG:   host = X/X
> STARTUP_MSG:   args = []
> STARTUP_MSG:   version = 3.2.0
> (snip)
> {code}
> Web browser console outputs an error like below
> {code:java}
> TypeError: "c[0].match(...) is null"
>   i http://localhost:8088/ui2/assets/vendor.js:5598:40973
>   l http://localhost:8088/ui2/assets/vendor.js:5598:41338
>   p http://localhost:8088/ui2/assets/vendor.js:5598:42035
>   q http://localhost:8088/ui2/assets/vendor.js:5598:42235
>   getDefaultTimezone http://localhost:8088/ui2/assets/yarn-ui.js:378:464
>   convertTimestampWithTz 
> http://localhost:8088/ui2/assets/yarn-ui.js:379:220
>   timeStampToDate http://localhost:8088/ui2/assets/yarn-ui.js:360:80
>   dateFormatter http://localhost:8088/ui2/assets/yarn-ui.js:177:1011
>   compute http://localhost:8088/ui2/assets/vendor.js:1052:780
>   value http://localhost:8088/ui2/assets/vendor.js:1528:12
>   read http://localhost:8088/ui2/assets/vendor.js:1544:58
>   readArray http://localhost:8088/ui2/assets/vendor.js:1545:110
>   compute http://localhost:8088/ui2/assets/vendor.js:1553:317
>   value http://localhost:8088/ui2/assets/vendor.js:1528:12
>   read http://localhost:8088/ui2/assets/vendor.js:1544:58
>   getValue http://localhost:8088/ui2/assets/vendor.js:907:329
>   attribute http://localhost:8088/ui2/assets/vendor.js:2547:54
>   attribute http://localhost:8088/ui2/assets/vendor.js:2575:623
>   populateNodes http://localhost:8088/ui2/assets/vendor.js:2610:334
>   render http://localhost:8088/ui2/assets/vendor.js:2605:265
>   render http://localhost:8088/ui2/assets/vendor.js:2579:122
>   yieldTemplate http://localhost:8088/ui2/assets/vendor.js:2479:155
>   ifUnless http://localhost:8088/ui2/assets/vendor.js:871:85
>   ifHelper http://localhost:8088/ui2/assets/vendor.js:869:524
>   compute http://localhost:8088/ui2/assets/vendor.js:1051:500
>   value http://localhost:8088/ui2/assets/vendor.js:1528:12
>   invokeHelper http://localhost:8088/ui2/assets/vendor.js:914:14
>   continueBlock http://localhost:8088/ui2/assets/vendor.js:2504:214
>   renderAndCleanup http://localhost:8088/ui2/assets/vendor.js:2661:189
>   hostBlock http://localhost:8088/ui2/assets/vendor.js:2505:150
>   continueBlock http://localhost:8088/ui2/assets/vendor.js:2504:83
>   block http://localhost:8088/ui2/assets/vendor.js:2503:1
>   block http://localhost:8088/ui2/assets/vendor.js:2572:288
>   populateNodes http://localhost:8088/ui2/assets/vendor.js:2610:34
>   render http://localhost:8088/ui2/assets/vendor.js:2605:265
>   render http://localhost:8088/ui2/assets/vendor.js:2579:122
>   _firstRender http://localhost:8088/ui2/assets/vendor.js:2658:245
>   renderAndCleanup http://localhost:8088/ui2/assets/vendor.js:2661:189
>

[jira] [Updated] (YARN-9248) RMContainerImpl:Invalid event: ACQUIRED at KILLED



 [ 
https://issues.apache.org/jira/browse/YARN-9248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

lujie updated YARN-9248:

Attachment: YARN-9248_5.patch

> RMContainerImpl:Invalid event: ACQUIRED at KILLED
> -
>
> Key: YARN-9248
> URL: https://issues.apache.org/jira/browse/YARN-9248
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: lujie
>Assignee: lujie
>Priority: Major
> Attachments: YARN-9248_1.patch, YARN-9248_2.patch, YARN-9248_3.patch, 
> YARN-9248_4.patch, YARN-9248_5.patch
>
>
> {code:java}
> 2019-01-29 11:46:53,596 ERROR 
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: 
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event: 
> ACQUIRED at KILLED
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:475)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:67)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.handleNewContainers(OpportunisticContainerAllocatorAMService.java:351)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.access$100(OpportunisticContainerAllocatorAMService.java:94)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService$OpportunisticAMSProcessor.allocate(OpportunisticContainerAllocatorAMService.java:197)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:424)
> at 
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at 
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:530)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:878)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2830)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9240) YARN UI 2 footer shows the datetime in different timezone

2019-01-30 Thread Masahiro Tanaka (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756239#comment-16756239
 ] 

Masahiro Tanaka commented on YARN-9240:
---

Hi Akhil PB

> So YARN-8747 patch fixes the this issue, right?

Yes. I built Hadoop from the latest source code which includes YARN-8747, and 
confirmed the problem was fixed.

According to [https://github.com/moment/moment-timezone/releases/tag/0.5.1] ,

"moment/moment-timezone" v0.5.1 includes the PR I mentioned 
[above|https://github.com/moment/moment-timezone/pull/302].

> YARN UI 2 footer shows the datetime in different timezone
> -
>
> Key: YARN-9240
> URL: https://issues.apache.org/jira/browse/YARN-9240
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn-ui-v2
>Affects Versions: 3.2.0
> Environment: Windows 10/Firefox 64
> {code}
>   (new Date()).toTimeString()
>   => "09:11:16 GMT+0900 (日本標準時)"
> {code}
>Reporter: Masahiro Tanaka
>Assignee: Masahiro Tanaka
>Priority: Minor
>
> YARN Web UI 2 footer shows the started time about "2019-01-25 16:39" even if 
> the ResourceManager started at "2019-01-26 00:39:34"(UTC), and my PC's 
> localtime is JST.(+09:00 GMT)
> ResourceManager log is below: (Sever time is set as UTC). 
> {code}
> 2019-01-26 00:39:34,619 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: STARTUP_MSG:
> /
> STARTUP_MSG: Starting ResourceManager
> STARTUP_MSG:   host = X/X
> STARTUP_MSG:   args = []
> STARTUP_MSG:   version = 3.2.0
> (snip)
> {code}
> Web browser console outputs an error like below
> {code:java}
> TypeError: "c[0].match(...) is null"
>   i http://localhost:8088/ui2/assets/vendor.js:5598:40973
>   l http://localhost:8088/ui2/assets/vendor.js:5598:41338
>   p http://localhost:8088/ui2/assets/vendor.js:5598:42035
>   q http://localhost:8088/ui2/assets/vendor.js:5598:42235
>   getDefaultTimezone http://localhost:8088/ui2/assets/yarn-ui.js:378:464
>   convertTimestampWithTz 
> http://localhost:8088/ui2/assets/yarn-ui.js:379:220
>   timeStampToDate http://localhost:8088/ui2/assets/yarn-ui.js:360:80
>   dateFormatter http://localhost:8088/ui2/assets/yarn-ui.js:177:1011
>   compute http://localhost:8088/ui2/assets/vendor.js:1052:780
>   value http://localhost:8088/ui2/assets/vendor.js:1528:12
>   read http://localhost:8088/ui2/assets/vendor.js:1544:58
>   readArray http://localhost:8088/ui2/assets/vendor.js:1545:110
>   compute http://localhost:8088/ui2/assets/vendor.js:1553:317
>   value http://localhost:8088/ui2/assets/vendor.js:1528:12
>   read http://localhost:8088/ui2/assets/vendor.js:1544:58
>   getValue http://localhost:8088/ui2/assets/vendor.js:907:329
>   attribute http://localhost:8088/ui2/assets/vendor.js:2547:54
>   attribute http://localhost:8088/ui2/assets/vendor.js:2575:623
>   populateNodes http://localhost:8088/ui2/assets/vendor.js:2610:334
>   render http://localhost:8088/ui2/assets/vendor.js:2605:265
>   render http://localhost:8088/ui2/assets/vendor.js:2579:122
>   yieldTemplate http://localhost:8088/ui2/assets/vendor.js:2479:155
>   ifUnless http://localhost:8088/ui2/assets/vendor.js:871:85
>   ifHelper http://localhost:8088/ui2/assets/vendor.js:869:524
>   compute http://localhost:8088/ui2/assets/vendor.js:1051:500
>   value http://localhost:8088/ui2/assets/vendor.js:1528:12
>   invokeHelper http://localhost:8088/ui2/assets/vendor.js:914:14
>   continueBlock http://localhost:8088/ui2/assets/vendor.js:2504:214
>   renderAndCleanup http://localhost:8088/ui2/assets/vendor.js:2661:189
>   hostBlock http://localhost:8088/ui2/assets/vendor.js:2505:150
>   continueBlock http://localhost:8088/ui2/assets/vendor.js:2504:83
>   block http://localhost:8088/ui2/assets/vendor.js:2503:1
>   block http://localhost:8088/ui2/assets/vendor.js:2572:288
>   populateNodes http://localhost:8088/ui2/assets/vendor.js:2610:34
>   render http://localhost:8088/ui2/assets/vendor.js:2605:265
>   render http://localhost:8088/ui2/assets/vendor.js:2579:122
>   _firstRender http://localhost:8088/ui2/assets/vendor.js:2658:245
>   renderAndCleanup http://localhost:8088/ui2/assets/vendor.js:2661:189
>   _firstRender http://localhost:8088/ui2/assets/vendor.js:2658:55
>   invoke http://localhost:8088/ui2/assets/vendor.js:2657:203
>   yieldKeyword http://localhost:8088/ui2/assets/vendor.js:991:888
>   handleKeyword http://localhost:8088/ui2/assets/vendor.js:2512:40
>   handleRedirect http://localhost:8088/ui2/assets/vendor.js:2509:4
>   inline http://localhost:8088/ui2/assets/vendor.js:2528:62
>   content

[jira] [Commented] (YARN-9060) [YARN-8851] Phase 1 - Support device isolation and use the Nvidia GPU plugin as an example



[ 
https://issues.apache.org/jira/browse/YARN-9060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756238#comment-16756238
 ] 

Zhankun Tang commented on YARN-9060:


[~cheersyang] , Thanks for the review!

For point 1, added the check.

For point 2, changed the log message.

For point 3, removed the unneeded code.

For point 4, catch the number parse exception. And in theory, all exception 
from the plugin method will be caught by the framework.

For point 5, Changed the plugin name to "NvidiaGPUPluginForRuntimeV2" and 
clarify the name in the comments that it supports Nvidia container runtime v2 
for Docker and non-Docker container.

For the other points, done.

> [YARN-8851] Phase 1 - Support device isolation and use the Nvidia GPU plugin 
> as an example
> --
>
> Key: YARN-9060
> URL: https://issues.apache.org/jira/browse/YARN-9060
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
>Priority: Major
> Attachments: YARN-9060-trunk.001.patch, YARN-9060-trunk.002.patch, 
> YARN-9060-trunk.003.patch, YARN-9060-trunk.004.patch, 
> YARN-9060-trunk.005.patch, YARN-9060-trunk.006.patch, 
> YARN-9060-trunk.007.patch, YARN-9060-trunk.008.patch, 
> YARN-9060-trunk.009.patch, YARN-9060-trunk.010.patch, 
> YARN-9060-trunk.011.patch, YARN-9060-trunk.012.patch, 
> YARN-9060-trunk.013.patch
>
>
> Due to the cgroups v1 implementation policy in linux kernel, we cannot update 
> the value of the device cgroups controller unless we have the root permission 
> ([here|https://github.com/torvalds/linux/blob/6f0d349d922ba44e4348a17a78ea51b7135965b1/security/device_cgroup.c#L604]).
>  So we need to support this in container-executor for Java layer to invoke.
> This Jira will have three parts:
>  # native c-e module
>  # Java layer code to isolate devices for container (docker and non-docker)
>  # A sample Nvidia GPU plugin



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9060) [YARN-8851] Phase 1 - Support device isolation and use the Nvidia GPU plugin as an example



 [ 
https://issues.apache.org/jira/browse/YARN-9060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhankun Tang updated YARN-9060:
---
Attachment: YARN-9060-trunk.013.patch

> [YARN-8851] Phase 1 - Support device isolation and use the Nvidia GPU plugin 
> as an example
> --
>
> Key: YARN-9060
> URL: https://issues.apache.org/jira/browse/YARN-9060
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
>Priority: Major
> Attachments: YARN-9060-trunk.001.patch, YARN-9060-trunk.002.patch, 
> YARN-9060-trunk.003.patch, YARN-9060-trunk.004.patch, 
> YARN-9060-trunk.005.patch, YARN-9060-trunk.006.patch, 
> YARN-9060-trunk.007.patch, YARN-9060-trunk.008.patch, 
> YARN-9060-trunk.009.patch, YARN-9060-trunk.010.patch, 
> YARN-9060-trunk.011.patch, YARN-9060-trunk.012.patch, 
> YARN-9060-trunk.013.patch
>
>
> Due to the cgroups v1 implementation policy in linux kernel, we cannot update 
> the value of the device cgroups controller unless we have the root permission 
> ([here|https://github.com/torvalds/linux/blob/6f0d349d922ba44e4348a17a78ea51b7135965b1/security/device_cgroup.c#L604]).
>  So we need to support this in container-executor for Java layer to invoke.
> This Jira will have three parts:
>  # native c-e module
>  # Java layer code to isolate devices for container (docker and non-docker)
>  # A sample Nvidia GPU plugin



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9230) Write a go hdfs driver for Docker Registry

2019-01-30 Thread Shane Kumpf (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756164#comment-16756164
 ] 

Shane Kumpf commented on YARN-9230:
---

Thanks for taking another look at the library, [~eyang]. It sounds like some 
improvements have been made, but gaps still exist. The security gaps in the go 
HDFS client is likely a non-starter for many. I'm inclined to agree that a 
registry storage driver for HDFS won't be implemented until this gap is fixed 
and fixing the gap is outside of the scope of YARN. The issue could be reopened 
if security is properly implemented.

> Write a go hdfs driver for Docker Registry
> --
>
> Key: YARN-9230
> URL: https://issues.apache.org/jira/browse/YARN-9230
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Eric Yang
>Priority: Major
>
> A number of people have developed go client library for HDFS.  [Example 
> 1|https://medium.com/sqooba/making-hdfs-a-hundred-times-faster-ac75b8b5e0b4] 
> [Example 2|https://github.com/colinmarc/hdfs].
> This can be enhanced into a real storage driver for Docker registry.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9251) Build failure for -Dhbase.profile=2.0

2019-01-30 Thread Hudson (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756150#comment-16756150
 ] 

Hudson commented on YARN-9251:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15853 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/15853/])
YARN-9251. Build failure for -Dhbase.profile=2.0. Contributed by Rohith 
(aajisaka: rev a3a9ae3cea077c55113ae722e69f09b07c81cc27)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/pom.xml


> Build failure for -Dhbase.profile=2.0
> -
>
> Key: YARN-9251
> URL: https://issues.apache.org/jira/browse/YARN-9251
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Blocker
> Fix For: 3.3.0
>
> Attachments: HADOOP-16088.01.patch
>
>
> Post HADOOP-14178, hadoop build failure due to incorrect pom.xml. 
> {noformat}
> HW12723:hadoop rsharmaks$ mvn clean install -DskipTests -DskipShade 
> -Dhbase.profile=2.0
> [INFO] Scanning for projects...
> [ERROR] [ERROR] Some problems were encountered while processing the POMs:
> [ERROR] 'dependencies.dependency.version' for org.mockito:mockito-all:jar is 
> missing. @ line 485, column 21
>  @
> [ERROR] The build could not read 1 project -> [Help 1]
> [ERROR]
> [ERROR]   The project 
> org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-tests:3.3.0-SNAPSHOT
>  
> (/Users/rsharmaks/Repos/Apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/pom.xml)
>  has 1 error
> [ERROR] 'dependencies.dependency.version' for org.mockito:mockito-all:jar 
> is missing. @ line 485, column 21
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please 
> read the following articles:
> [ERROR] [Help 1] 
> http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
> {noformat}
> cc:/ [~ajisakaa]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Issue Comment Deleted] (YARN-9248) RMContainerImpl:Invalid event: ACQUIRED at KILLED



 [ 
https://issues.apache.org/jira/browse/YARN-9248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

lujie updated YARN-9248:

Comment: was deleted

(was: The UT failure are due to 
TestRMAppAttemptTransitions#testContainerRemovedBeforeAllocate.

Patch-1 does't trigger the failure, patch-4 only change code style based on 
patch-1, but trigger the UT failue, My local test shows that both

TestRMAppAttemptTransitions# testContainerRemovedBeforeAllocate and  
TestRMContainerImpl#testContainerAcquiredAtKilledwork well, so I don't think 
the faiure is  caused by this patch.

Hope someone  review it and make sure it.)

> RMContainerImpl:Invalid event: ACQUIRED at KILLED
> -
>
> Key: YARN-9248
> URL: https://issues.apache.org/jira/browse/YARN-9248
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: lujie
>Assignee: lujie
>Priority: Major
> Attachments: YARN-9248_1.patch, YARN-9248_2.patch, YARN-9248_3.patch, 
> YARN-9248_4.patch
>
>
> {code:java}
> 2019-01-29 11:46:53,596 ERROR 
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: 
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event: 
> ACQUIRED at KILLED
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:475)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:67)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.handleNewContainers(OpportunisticContainerAllocatorAMService.java:351)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.access$100(OpportunisticContainerAllocatorAMService.java:94)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService$OpportunisticAMSProcessor.allocate(OpportunisticContainerAllocatorAMService.java:197)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:424)
> at 
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at 
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:530)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:878)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2830)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-9248) RMContainerImpl:Invalid event: ACQUIRED at KILLED



[ 
https://issues.apache.org/jira/browse/YARN-9248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756138#comment-16756138
 ] 

lujie edited comment on YARN-9248 at 1/30/19 2:01 PM:
--

The UT failure are due to 
TestRMAppAttemptTransitions#testContainerRemovedBeforeAllocate.

Patch-1 does't trigger the failure, patch-4 only change code style based on 
patch-1, but trigger the UT failue, My local test shows that both

TestRMAppAttemptTransitions# testContainerRemovedBeforeAllocate and  
TestRMContainerImpl#testContainerAcquiredAtKilledwork well, so I don't think 
the faiure is  caused by this patch.

Hope someone  review it and make sure it.


was (Author: xiaoheipangzi):
The UT failure are due to 
TestRMAppAttemptTransitions#testContainerRemovedBeforeAllocate.

Patch-1 does't trigger the failure, patch-2 ~ 4 only change code style based on 
patch-1, but trigger the UT failue, My local test shows that both

TestRMAppAttemptTransitions# testContainerRemovedBeforeAllocate and  
TestRMContainerImpl#testContainerAcquiredAtKilledwork well, so I don't think 
the faiure is  caused by this patch.

Hope someone  review it and make sure it.

> RMContainerImpl:Invalid event: ACQUIRED at KILLED
> -
>
> Key: YARN-9248
> URL: https://issues.apache.org/jira/browse/YARN-9248
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: lujie
>Assignee: lujie
>Priority: Major
> Attachments: YARN-9248_1.patch, YARN-9248_2.patch, YARN-9248_3.patch, 
> YARN-9248_4.patch
>
>
> {code:java}
> 2019-01-29 11:46:53,596 ERROR 
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: 
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event: 
> ACQUIRED at KILLED
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:475)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:67)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.handleNewContainers(OpportunisticContainerAllocatorAMService.java:351)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.access$100(OpportunisticContainerAllocatorAMService.java:94)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService$OpportunisticAMSProcessor.allocate(OpportunisticContainerAllocatorAMService.java:197)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:424)
> at 
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at 
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:530)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:878)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2830)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9248) RMContainerImpl:Invalid event: ACQUIRED at KILLED



[ 
https://issues.apache.org/jira/browse/YARN-9248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756138#comment-16756138
 ] 

lujie commented on YARN-9248:
-

The UT failure are due to 
TestRMAppAttemptTransitions#testContainerRemovedBeforeAllocate.

Patch-1 does't trigger the failure, patch-2 ~ 4 only change code style based on 
patch-1, but trigger the UT failue, My local test shows that both

TestRMAppAttemptTransitions# testContainerRemovedBeforeAllocate and  
TestRMContainerImpl#testContainerAcquiredAtKilledwork well, so I don't think 
the faiure is  caused by this patch.

Hope someone  review it and make sure it.

> RMContainerImpl:Invalid event: ACQUIRED at KILLED
> -
>
> Key: YARN-9248
> URL: https://issues.apache.org/jira/browse/YARN-9248
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: lujie
>Assignee: lujie
>Priority: Major
> Attachments: YARN-9248_1.patch, YARN-9248_2.patch, YARN-9248_3.patch, 
> YARN-9248_4.patch
>
>
> {code:java}
> 2019-01-29 11:46:53,596 ERROR 
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: 
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event: 
> ACQUIRED at KILLED
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:475)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:67)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.handleNewContainers(OpportunisticContainerAllocatorAMService.java:351)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.access$100(OpportunisticContainerAllocatorAMService.java:94)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService$OpportunisticAMSProcessor.allocate(OpportunisticContainerAllocatorAMService.java:197)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:424)
> at 
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at 
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:530)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:878)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2830)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Moved] (YARN-9251) Build failure for -Dhbase.profile=2.0

2019-01-30 Thread Akira Ajisaka (JIRA)



 [ 
https://issues.apache.org/jira/browse/YARN-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka moved HADOOP-16088 to YARN-9251:
--

Target Version/s: 3.3.0  (was: 3.3.0)
 Key: YARN-9251  (was: HADOOP-16088)
 Project: Hadoop YARN  (was: Hadoop Common)

> Build failure for -Dhbase.profile=2.0
> -
>
> Key: YARN-9251
> URL: https://issues.apache.org/jira/browse/YARN-9251
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Blocker
> Attachments: HADOOP-16088.01.patch
>
>
> Post HADOOP-14178, hadoop build failure due to incorrect pom.xml. 
> {noformat}
> HW12723:hadoop rsharmaks$ mvn clean install -DskipTests -DskipShade 
> -Dhbase.profile=2.0
> [INFO] Scanning for projects...
> [ERROR] [ERROR] Some problems were encountered while processing the POMs:
> [ERROR] 'dependencies.dependency.version' for org.mockito:mockito-all:jar is 
> missing. @ line 485, column 21
>  @
> [ERROR] The build could not read 1 project -> [Help 1]
> [ERROR]
> [ERROR]   The project 
> org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-tests:3.3.0-SNAPSHOT
>  
> (/Users/rsharmaks/Repos/Apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/pom.xml)
>  has 1 error
> [ERROR] 'dependencies.dependency.version' for org.mockito:mockito-all:jar 
> is missing. @ line 485, column 21
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please 
> read the following articles:
> [ERROR] [Help 1] 
> http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
> {noformat}
> cc:/ [~ajisakaa]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9206) RMServerUtils does not count SHUTDOWN as an accepted state

2019-01-30 Thread Kuhu Shukla (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756075#comment-16756075
 ] 

Kuhu Shukla commented on YARN-9206:
---

Thank you [~sunilg], will update patch shortly.

> RMServerUtils does not count SHUTDOWN as an accepted state
> --
>
> Key: YARN-9206
> URL: https://issues.apache.org/jira/browse/YARN-9206
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 3.0.3
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
>Priority: Major
> Attachments: YARN-9206.001.patch, YARN-9206.002.patch, 
> YARN-9206.003.patch
>
>
> {code}
> if (acceptedStates.contains(NodeState.DECOMMISSIONED) ||
> acceptedStates.contains(NodeState.LOST) ||
> acceptedStates.contains(NodeState.REBOOTED)) {
>   for (RMNode rmNode : context.getInactiveRMNodes().values()) {
> if ((rmNode != null) && acceptedStates.contains(rmNode.getState())) {
>   results.add(rmNode);
> }
>   }
> }
> return results;
>   }
> {code}
> This should include SHUTDOWN state as they are inactive too. This method is 
> used for node reports and such so might be useful to account for them as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9250) hadoop-yarn-server-nodemanager build failed: make failed with error code 2



 [ 
https://issues.apache.org/jira/browse/YARN-9250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

charlie mao updated YARN-9250:
--
Description: 
when i compile hadoop-3.2.0 release,i encounter following errors：

[ERROR] Failed to execute goal 
org.apache.hadoop:hadoop-maven-plugins:3.2.0:cmake-compile (cmake-compile) on 
project hadoop-yarn-server-nodemanager: make failed with error code 2 -> [Help 
1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal 
org.apache.hadoop:hadoop-maven-plugins:3.2.0:cmake-compile (cmake-compile) on 
project hadoop-yarn-server-nodemanager: make failed with error code 2
 at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
 at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
 at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
 at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
 at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
 at 
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
 at 
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
 at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
 at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
 at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
 at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
 at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
 at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at 
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
 at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
 at 
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
 at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: make failed with 
error code 2
 at 
org.apache.hadoop.maven.plugin.cmakebuilder.CompileMojo.runMake(CompileMojo.java:231)
 at 
org.apache.hadoop.maven.plugin.cmakebuilder.CompileMojo.execute(CompileMojo.java:98)
 at 
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
 at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
 ... 20 more
[ERROR]
[ERROR]
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn  -rf :hadoop-yarn-server-nodemanager

 

my compiling environments：

jdk .8.0_181.8.0_181

maven:3.3.9(3.6.0)

cmake version 3.12.0



  was:
when i compile hadoop-3.2.0 release,i encounter following errors：

[ERROR] Failed to execute goal 
org.apache.hadoop:hadoop-maven-plugins:3.2.0:cmake-compile (cmake-compile) on 
project hadoop-yarn-server-nodemanager: make failed with error code 2 -> [Help 
1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal 
org.apache.hadoop:hadoop-maven-plugins:3.2.0:cmake-compile (cmake-compile) on 
project hadoop-yarn-server-nodemanager: make failed with error code 2
 at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
 at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
 at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
 at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
 at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
 at 
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
 at 
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
 at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
 at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
 at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
 at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
 at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
 at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at

[jira] [Created] (YARN-9250) hadoop-yarn-server-nodemanager build fail make failed with error code 2

charlie mao created YARN-9250:
-

 Summary: hadoop-yarn-server-nodemanager build fail make failed 
with error code 2
 Key: YARN-9250
 URL: https://issues.apache.org/jira/browse/YARN-9250
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 3.2.0
Reporter: charlie mao


when i compile hadoop-3.2.0 release,i encounter following errors：

[ERROR] Failed to execute goal 
org.apache.hadoop:hadoop-maven-plugins:3.2.0:cmake-compile (cmake-compile) on 
project hadoop-yarn-server-nodemanager: make failed with error code 2 -> [Help 
1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal 
org.apache.hadoop:hadoop-maven-plugins:3.2.0:cmake-compile (cmake-compile) on 
project hadoop-yarn-server-nodemanager: make failed with error code 2
 at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
 at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
 at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
 at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
 at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
 at 
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
 at 
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
 at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
 at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
 at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
 at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
 at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
 at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at 
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
 at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
 at 
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
 at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: make failed with 
error code 2
 at 
org.apache.hadoop.maven.plugin.cmakebuilder.CompileMojo.runMake(CompileMojo.java:231)
 at 
org.apache.hadoop.maven.plugin.cmakebuilder.CompileMojo.execute(CompileMojo.java:98)
 at 
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
 at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
 ... 20 more
[ERROR]
[ERROR]
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn  -rf :hadoop-yarn-server-nodemanager

 

my compile environment：

jdk .8.0_181.8.0_181

maven:3.3.9(3.6.0)

cmake version 3.12.0





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9250) hadoop-yarn-server-nodemanager build failed: make failed with error code 2



 [ 
https://issues.apache.org/jira/browse/YARN-9250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

charlie mao updated YARN-9250:
--
Summary: hadoop-yarn-server-nodemanager build failed: make failed with 
error code 2  (was: hadoop-yarn-server-nodemanager build fail make failed with 
error code 2)

> hadoop-yarn-server-nodemanager build failed: make failed with error code 2
> --
>
> Key: YARN-9250
> URL: https://issues.apache.org/jira/browse/YARN-9250
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 3.2.0
>Reporter: charlie mao
>Priority: Blocker
>
> when i compile hadoop-3.2.0 release,i encounter following errors：
> [ERROR] Failed to execute goal 
> org.apache.hadoop:hadoop-maven-plugins:3.2.0:cmake-compile (cmake-compile) on 
> project hadoop-yarn-server-nodemanager: make failed with error code 2 -> 
> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
> goal org.apache.hadoop:hadoop-maven-plugins:3.2.0:cmake-compile 
> (cmake-compile) on project hadoop-yarn-server-nodemanager: make failed with 
> error code 2
>  at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
>  at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>  at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>  at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
>  at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
>  at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>  at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
>  at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
>  at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
>  at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
>  at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
>  at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
>  at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>  at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498)
>  at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
>  at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
>  at 
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
>  at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
> Caused by: org.apache.maven.plugin.MojoExecutionException: make failed with 
> error code 2
>  at 
> org.apache.hadoop.maven.plugin.cmakebuilder.CompileMojo.runMake(CompileMojo.java:231)
>  at 
> org.apache.hadoop.maven.plugin.cmakebuilder.CompileMojo.execute(CompileMojo.java:98)
>  at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
>  at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
>  ... 20 more
> [ERROR]
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please 
> read the following articles:
> [ERROR] [Help 1] 
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the 
> command
> [ERROR] mvn  -rf :hadoop-yarn-server-nodemanager
>  
> my compile environment：
> jdk .8.0_181.8.0_181
> maven:3.3.9(3.6.0)
> cmake version 3.12.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9250) hadoop-yarn-server-nodemanager build fail make failed with error code 2



[ 
https://issues.apache.org/jira/browse/YARN-9250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756037#comment-16756037
 ] 

charlie mao commented on YARN-9250:
---

i did this  on Mac os

> hadoop-yarn-server-nodemanager build fail make failed with error code 2
> ---
>
> Key: YARN-9250
> URL: https://issues.apache.org/jira/browse/YARN-9250
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 3.2.0
>Reporter: charlie mao
>Priority: Blocker
>
> when i compile hadoop-3.2.0 release,i encounter following errors：
> [ERROR] Failed to execute goal 
> org.apache.hadoop:hadoop-maven-plugins:3.2.0:cmake-compile (cmake-compile) on 
> project hadoop-yarn-server-nodemanager: make failed with error code 2 -> 
> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
> goal org.apache.hadoop:hadoop-maven-plugins:3.2.0:cmake-compile 
> (cmake-compile) on project hadoop-yarn-server-nodemanager: make failed with 
> error code 2
>  at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
>  at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>  at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>  at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
>  at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
>  at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>  at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
>  at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
>  at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
>  at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
>  at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
>  at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
>  at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>  at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498)
>  at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
>  at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
>  at 
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
>  at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
> Caused by: org.apache.maven.plugin.MojoExecutionException: make failed with 
> error code 2
>  at 
> org.apache.hadoop.maven.plugin.cmakebuilder.CompileMojo.runMake(CompileMojo.java:231)
>  at 
> org.apache.hadoop.maven.plugin.cmakebuilder.CompileMojo.execute(CompileMojo.java:98)
>  at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
>  at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
>  ... 20 more
> [ERROR]
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please 
> read the following articles:
> [ERROR] [Help 1] 
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the 
> command
> [ERROR] mvn  -rf :hadoop-yarn-server-nodemanager
>  
> my compile environment：
> jdk .8.0_181.8.0_181
> maven:3.3.9(3.6.0)
> cmake version 3.12.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9240) YARN UI 2 footer shows the datetime in different timezone

2019-01-30 Thread Akhil PB (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756000#comment-16756000
 ] 

Akhil PB commented on YARN-9240:


Hi [~masatana] 

So YARN-8747 patch fixes the issue you mentioned above, right? Because I could 
not figure out the moment-timezone fix version from the PR given above.

> YARN UI 2 footer shows the datetime in different timezone
> -
>
> Key: YARN-9240
> URL: https://issues.apache.org/jira/browse/YARN-9240
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn-ui-v2
>Affects Versions: 3.2.0
> Environment: Windows 10/Firefox 64
> {code}
>   (new Date()).toTimeString()
>   => "09:11:16 GMT+0900 (日本標準時)"
> {code}
>Reporter: Masahiro Tanaka
>Assignee: Masahiro Tanaka
>Priority: Minor
>
> YARN Web UI 2 footer shows the started time about "2019-01-25 16:39" even if 
> the ResourceManager started at "2019-01-26 00:39:34"(UTC), and my PC's 
> localtime is JST.(+09:00 GMT)
> ResourceManager log is below: (Sever time is set as UTC). 
> {code}
> 2019-01-26 00:39:34,619 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: STARTUP_MSG:
> /
> STARTUP_MSG: Starting ResourceManager
> STARTUP_MSG:   host = X/X
> STARTUP_MSG:   args = []
> STARTUP_MSG:   version = 3.2.0
> (snip)
> {code}
> Web browser console outputs an error like below
> {code:java}
> TypeError: "c[0].match(...) is null"
>   i http://localhost:8088/ui2/assets/vendor.js:5598:40973
>   l http://localhost:8088/ui2/assets/vendor.js:5598:41338
>   p http://localhost:8088/ui2/assets/vendor.js:5598:42035
>   q http://localhost:8088/ui2/assets/vendor.js:5598:42235
>   getDefaultTimezone http://localhost:8088/ui2/assets/yarn-ui.js:378:464
>   convertTimestampWithTz 
> http://localhost:8088/ui2/assets/yarn-ui.js:379:220
>   timeStampToDate http://localhost:8088/ui2/assets/yarn-ui.js:360:80
>   dateFormatter http://localhost:8088/ui2/assets/yarn-ui.js:177:1011
>   compute http://localhost:8088/ui2/assets/vendor.js:1052:780
>   value http://localhost:8088/ui2/assets/vendor.js:1528:12
>   read http://localhost:8088/ui2/assets/vendor.js:1544:58
>   readArray http://localhost:8088/ui2/assets/vendor.js:1545:110
>   compute http://localhost:8088/ui2/assets/vendor.js:1553:317
>   value http://localhost:8088/ui2/assets/vendor.js:1528:12
>   read http://localhost:8088/ui2/assets/vendor.js:1544:58
>   getValue http://localhost:8088/ui2/assets/vendor.js:907:329
>   attribute http://localhost:8088/ui2/assets/vendor.js:2547:54
>   attribute http://localhost:8088/ui2/assets/vendor.js:2575:623
>   populateNodes http://localhost:8088/ui2/assets/vendor.js:2610:334
>   render http://localhost:8088/ui2/assets/vendor.js:2605:265
>   render http://localhost:8088/ui2/assets/vendor.js:2579:122
>   yieldTemplate http://localhost:8088/ui2/assets/vendor.js:2479:155
>   ifUnless http://localhost:8088/ui2/assets/vendor.js:871:85
>   ifHelper http://localhost:8088/ui2/assets/vendor.js:869:524
>   compute http://localhost:8088/ui2/assets/vendor.js:1051:500
>   value http://localhost:8088/ui2/assets/vendor.js:1528:12
>   invokeHelper http://localhost:8088/ui2/assets/vendor.js:914:14
>   continueBlock http://localhost:8088/ui2/assets/vendor.js:2504:214
>   renderAndCleanup http://localhost:8088/ui2/assets/vendor.js:2661:189
>   hostBlock http://localhost:8088/ui2/assets/vendor.js:2505:150
>   continueBlock http://localhost:8088/ui2/assets/vendor.js:2504:83
>   block http://localhost:8088/ui2/assets/vendor.js:2503:1
>   block http://localhost:8088/ui2/assets/vendor.js:2572:288
>   populateNodes http://localhost:8088/ui2/assets/vendor.js:2610:34
>   render http://localhost:8088/ui2/assets/vendor.js:2605:265
>   render http://localhost:8088/ui2/assets/vendor.js:2579:122
>   _firstRender http://localhost:8088/ui2/assets/vendor.js:2658:245
>   renderAndCleanup http://localhost:8088/ui2/assets/vendor.js:2661:189
>   _firstRender http://localhost:8088/ui2/assets/vendor.js:2658:55
>   invoke http://localhost:8088/ui2/assets/vendor.js:2657:203
>   yieldKeyword http://localhost:8088/ui2/assets/vendor.js:991:888
>   handleKeyword http://localhost:8088/ui2/assets/vendor.js:2512:40
>   handleRedirect http://localhost:8088/ui2/assets/vendor.js:2509:4
>   inline http://localhost:8088/ui2/assets/vendor.js:2528:62
>   content http://localhost:8088/ui2/assets/vendor.js:2572:903
>   populateNodes http://localhost:8088/ui2/assets/vendor.js:2610:181
>   render http://localhost:8088/ui2/assets/vendor.js:2605:265
>   render

[jira] [Comment Edited] (YARN-9240) YARN UI 2 footer shows the datetime in different timezone

2019-01-30 Thread Akhil PB (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756000#comment-16756000
 ] 

Akhil PB edited comment on YARN-9240 at 1/30/19 11:25 AM:
--

Hi [~masatana] 

So YARN-8747 patch fixes the this issue, right? Because I could not figure out 
the moment-timezone fix version from the PR given above. If the issue is fixed 
in the version > 0.5.1, then we may need to update the moment-timezone. 
YARN-8747 updates from 0.5.0 to 0.5.1.


was (Author: akhilpb):
Hi [~masatana] 

So YARN-8747 patch fixes the issue you mentioned above, right? Because I could 
not figure out the moment-timezone fix version from the PR given above.

> YARN UI 2 footer shows the datetime in different timezone
> -
>
> Key: YARN-9240
> URL: https://issues.apache.org/jira/browse/YARN-9240
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn-ui-v2
>Affects Versions: 3.2.0
> Environment: Windows 10/Firefox 64
> {code}
>   (new Date()).toTimeString()
>   => "09:11:16 GMT+0900 (日本標準時)"
> {code}
>Reporter: Masahiro Tanaka
>Assignee: Masahiro Tanaka
>Priority: Minor
>
> YARN Web UI 2 footer shows the started time about "2019-01-25 16:39" even if 
> the ResourceManager started at "2019-01-26 00:39:34"(UTC), and my PC's 
> localtime is JST.(+09:00 GMT)
> ResourceManager log is below: (Sever time is set as UTC). 
> {code}
> 2019-01-26 00:39:34,619 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: STARTUP_MSG:
> /
> STARTUP_MSG: Starting ResourceManager
> STARTUP_MSG:   host = X/X
> STARTUP_MSG:   args = []
> STARTUP_MSG:   version = 3.2.0
> (snip)
> {code}
> Web browser console outputs an error like below
> {code:java}
> TypeError: "c[0].match(...) is null"
>   i http://localhost:8088/ui2/assets/vendor.js:5598:40973
>   l http://localhost:8088/ui2/assets/vendor.js:5598:41338
>   p http://localhost:8088/ui2/assets/vendor.js:5598:42035
>   q http://localhost:8088/ui2/assets/vendor.js:5598:42235
>   getDefaultTimezone http://localhost:8088/ui2/assets/yarn-ui.js:378:464
>   convertTimestampWithTz 
> http://localhost:8088/ui2/assets/yarn-ui.js:379:220
>   timeStampToDate http://localhost:8088/ui2/assets/yarn-ui.js:360:80
>   dateFormatter http://localhost:8088/ui2/assets/yarn-ui.js:177:1011
>   compute http://localhost:8088/ui2/assets/vendor.js:1052:780
>   value http://localhost:8088/ui2/assets/vendor.js:1528:12
>   read http://localhost:8088/ui2/assets/vendor.js:1544:58
>   readArray http://localhost:8088/ui2/assets/vendor.js:1545:110
>   compute http://localhost:8088/ui2/assets/vendor.js:1553:317
>   value http://localhost:8088/ui2/assets/vendor.js:1528:12
>   read http://localhost:8088/ui2/assets/vendor.js:1544:58
>   getValue http://localhost:8088/ui2/assets/vendor.js:907:329
>   attribute http://localhost:8088/ui2/assets/vendor.js:2547:54
>   attribute http://localhost:8088/ui2/assets/vendor.js:2575:623
>   populateNodes http://localhost:8088/ui2/assets/vendor.js:2610:334
>   render http://localhost:8088/ui2/assets/vendor.js:2605:265
>   render http://localhost:8088/ui2/assets/vendor.js:2579:122
>   yieldTemplate http://localhost:8088/ui2/assets/vendor.js:2479:155
>   ifUnless http://localhost:8088/ui2/assets/vendor.js:871:85
>   ifHelper http://localhost:8088/ui2/assets/vendor.js:869:524
>   compute http://localhost:8088/ui2/assets/vendor.js:1051:500
>   value http://localhost:8088/ui2/assets/vendor.js:1528:12
>   invokeHelper http://localhost:8088/ui2/assets/vendor.js:914:14
>   continueBlock http://localhost:8088/ui2/assets/vendor.js:2504:214
>   renderAndCleanup http://localhost:8088/ui2/assets/vendor.js:2661:189
>   hostBlock http://localhost:8088/ui2/assets/vendor.js:2505:150
>   continueBlock http://localhost:8088/ui2/assets/vendor.js:2504:83
>   block http://localhost:8088/ui2/assets/vendor.js:2503:1
>   block http://localhost:8088/ui2/assets/vendor.js:2572:288
>   populateNodes http://localhost:8088/ui2/assets/vendor.js:2610:34
>   render http://localhost:8088/ui2/assets/vendor.js:2605:265
>   render http://localhost:8088/ui2/assets/vendor.js:2579:122
>   _firstRender http://localhost:8088/ui2/assets/vendor.js:2658:245
>   renderAndCleanup http://localhost:8088/ui2/assets/vendor.js:2661:189
>   _firstRender http://localhost:8088/ui2/assets/vendor.js:2658:55
>   invoke http://localhost:8088/ui2/assets/vendor.js:2657:203
>   yieldKeyword http://localhost:8088/ui2/assets/vendor.js:991:888
>   handleKeyword http://localhost:8088/ui2/assets/vendor.js:2512:40
>   handleRedirect

[jira] [Comment Edited] (YARN-9060) [YARN-8851] Phase 1 - Support device isolation and use the Nvidia GPU plugin as an example



[ 
https://issues.apache.org/jira/browse/YARN-9060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755941#comment-16755941
 ] 

Weiwei Yang edited comment on YARN-9060 at 1/30/19 10:02 AM:
-

 Hi [~tangzhankun]

Please take a look at my comments:

*NvidiaGPUPlugin*

1) NvidiaGPUPlugin#getDevices
{code:java}
for (String oneLine : lines) {
  String[] tokensEachLine = oneLine.split(",");
  String minorNumber = tokensEachLine[0].trim();
  String busId = tokensEachLine[1].trim();
  ...
}
{code}
The parsing code assumes the array len is 2, which can potentially throw out of 
index. Pls add a check here.

2) NvidiaGPUPlugin#searchBinary

We should use consistent logging, pls replace "script" with "binary".

and the log
{code:java}
if (!found) {
  LOG.error("No binary found in below path" + 
DEFAULT_BINARY_SEARCH_DIRS.toString());
  throw new Exception("No binary found for " + NvidiaGPUPlugin.class);
}
{code}
A more comprehensive logging can be something like

No binary found from env variable ENV_BINARY_PATH or path " + 
DEFAULT_BINARY_SEARCH_DIRS.toString()

3) Could u pls rename {{MyShellExecutor}} to something like 
{{NvidiaCommandExecutor}} 
{code:java}
if (yarnRuntime == YarnRuntimeType.RUNTIME_DEFAULT) {
  return null;
}
{code}
this can be removed

4) I think we need to catch the {{NumberFormateException}} here when parsing 
failed, and throw a IOException. Otherwise it might crash the NM.
{code:java}
output = Integer.toString(Integer.parseInt(strs[0], 16));
{code}
5)  Can we add version to the class implementation, in case in future there are 
multiple version implementations for GPU resource plugin?

*DeviceResourceDockerRuntimePluginImpl*

1)  DeviceResourceDockerRuntimePluginImpl#getCleanupDockerVolumesCommand
{code:java}
LOG.warn("Exception thrown onDeviceReleased of " + devicePlugin.getClass() + 
e.getMessage());
{code}
can be changed to
{code:java}
LOG.warn("Exception thrown onDeviceReleased of " + devicePlugin.getClass(), e);
{code}
*Misc*

Pls wrap LOG.DEBUG with
{code:java}
if (LOG.isDebugEnabled()){
  LOG.DEBUG("...")
}
{code}
Thanks


was (Author: cheersyang):
 

Hi [~tangzhankun]

Please take a look at my comments:

*NvidiaGPUPlugin*

1) NvidiaGPUPlugin#getDevices
{code:java}
for (String oneLine : lines) {
  String[] tokensEachLine = oneLine.split(",");
  String minorNumber = tokensEachLine[0].trim();
  String busId = tokensEachLine[1].trim();
  ...
}
{code}
The parsing code assumes the array len is 2, which can potentially throw out of 
index. Pls add a check here.

2) NvidiaGPUPlugin#searchBinary

We should use consistent logging, pls replace "script" with "binary".

and the log
{code:java}
if (!found) {
  LOG.error("No binary found in below path" + 
DEFAULT_BINARY_SEARCH_DIRS.toString());
  throw new Exception("No binary found for " + NvidiaGPUPlugin.class);
}
{code}
A more comprehensive logging can be something like

No binary found from env variable ENV_BINARY_PATH or path " + 
DEFAULT_BINARY_SEARCH_DIRS.toString()

3) Could u pls rename {{MyShellExecutor}} to something like 
{{NvidiaCommandExecutor}} 
{code:java}
if (yarnRuntime == YarnRuntimeType.RUNTIME_DEFAULT) {
  return null;
}
{code}
this can be removed

4) I think we need to catch the {{NumberFormateException}} here when parsing 
failed, and throw a IOException. Otherwise it might crash the NM.

{code}

output = Integer.toString(Integer.parseInt(strs[0], 16));

{code}

5)  Can we add version to the class implementation, in case in future there are 
multiple version implementations for GPU resource plugin?

*DeviceResourceDockerRuntimePluginImpl*

1)  DeviceResourceDockerRuntimePluginImpl#getCleanupDockerVolumesCommand

{code}

LOG.warn("Exception thrown onDeviceReleased of " + devicePlugin.getClass() + 
e.getMessage());

{code}

can be changed to

{code}

LOG.warn("Exception thrown onDeviceReleased of " + devicePlugin.getClass(), e);

{code}

*Misc*

Pls wrap LOG.DEBUG with

{code}

if (LOG.isDebugEnabled()){

  LOG.DEBUG("...")

}

{code}

Thanks

> [YARN-8851] Phase 1 - Support device isolation and use the Nvidia GPU plugin 
> as an example
> --
>
> Key: YARN-9060
> URL: https://issues.apache.org/jira/browse/YARN-9060
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
>Priority: Major
> Attachments: YARN-9060-trunk.001.patch, YARN-9060-trunk.002.patch, 
> YARN-9060-trunk.003.patch, YARN-9060-trunk.004.patch, 
> YARN-9060-trunk.005.patch, YARN-9060-trunk.006.patch, 
> YARN-9060-trunk.007.patch, YARN-9060-trunk.008.patch, 
> YARN-9060-trunk.009.patch, YARN-9060-trunk.010.patch, 
> YARN-9060-trunk.011.patch, YARN-9060-trunk.012.patch
>
>
> Due to the cgroups v1 implementation policy in linux kernel, we cannot update 
>

[jira] [Commented] (YARN-9060) [YARN-8851] Phase 1 - Support device isolation and use the Nvidia GPU plugin as an example