[jira] [Commented] (YARN-6189) Improve application status log message when RM restarted when app is in NEW state
[
https://issues.apache.org/jira/browse/YARN-6189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883957#comment-15883957
]
Junping Du commented on YARN-6189:
--
test failure should not be related.
> Improve application status log message when RM restarted when app is in NEW
> state
> -
>
> Key: YARN-6189
> URL: https://issues.apache.org/jira/browse/YARN-6189
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Yesha Vora
>Assignee: Junping Du
> Attachments: YARN-6189.patch, YARN-6189-v2.patch
>
>
> When RM restart/failover happens when application is in NEW state,
> application status command for that application prints below stacktrace.
> Improve exception message to less confusion to say something like:
> "application is not unknown, may be previous submission is
> not successful."
> {code}
> hrt_qa@:/root> yarn application -status application_1470379565464_0001
> 16/08/05 17:24:29 INFO impl.TimelineClientImpl: Timeline service address:
> https://hostxxx:8190/ws/v1/timeline/
> 16/08/05 17:24:30 INFO client.AHSProxy: Connecting to Application History
> server at hostxxx/xxx:10200
> 16/08/05 17:24:31 WARN retry.RetryInvocationHandler: Exception while invoking
> ApplicationClientProtocolPBClientImpl.getApplicationReport over rm1. Not
> retrying because try once and fail.
> org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application
> with id 'application_1470379565464_0001' doesn't exist in RM.
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:331)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:175)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:417)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:101)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:194)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
> at com.sun.proxy.$Proxy18.getApplicationReport(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:436)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.printApplicationReport(ApplicationCLI.java:481)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:160)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.main(ApplicationCLI.java:83)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException):
> Application with id 'application_1470379565464_0001' doesn't exist in RM.
> at
>
[jira] [Commented] (YARN-5280) Allow YARN containers to run with Java Security Manager
[
https://issues.apache.org/jira/browse/YARN-5280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883929#comment-15883929
]
Robert Kanter commented on YARN-5280:
-
[~gphillips], can you take a look at {{TestContainerManagerSecurity}}? It
seems to fail now with the 008 patch applied. Something must have changed in
the last week.
{noformat}
---
T E S T S
---
Picked up _JAVA_OPTIONS: -Djava.awt.headless=true
Running org.apache.hadoop.yarn.server.TestContainerManagerSecurity
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 41.505 sec <<<
FAILURE! - in org.apache.hadoop.yarn.server.TestContainerManagerSecurity
testContainerManager[1](org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
Time elapsed: 17.237 sec <<< ERROR!
java.lang.NullPointerException: null
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.waitForContainerToFinishOnNM(TestContainerManagerSecurity.java:399)
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testNMTokens(TestContainerManagerSecurity.java:342)
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testContainerManager(TestContainerManagerSecurity.java:159)
Results :
Tests in error:
TestContainerManagerSecurity.testContainerManager:159->testNMTokens:342->waitForContainerToFinishOnNM:399
NullPointer
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0
{noformat}
> Allow YARN containers to run with Java Security Manager
> ---
>
> Key: YARN-5280
> URL: https://issues.apache.org/jira/browse/YARN-5280
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: nodemanager, yarn
>Affects Versions: 2.6.4
>Reporter: Greg Phillips
>Assignee: Greg Phillips
>Priority: Minor
> Labels: oct16-medium
> Attachments: YARN-5280.001.patch, YARN-5280.002.patch,
> YARN-5280.003.patch, YARN-5280.004.patch, YARN-5280.005.patch,
> YARN-5280.006.patch, YARN-5280.007.patch, YARN-5280.008.patch,
> YARN-5280.patch, YARNContainerSandbox.pdf
>
>
> YARN applications have the ability to perform privileged actions which have
> the potential to add instability into the cluster. The Java Security Manager
> can be used to prevent users from running privileged actions while still
> allowing their core data processing use cases.
> Introduce a YARN flag which will allow a Hadoop administrator to enable the
> Java Security Manager for user code, while still providing complete
> permissions to core Hadoop libraries.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5946) Create YarnConfigurationStore interface and InMemoryConfigurationStore class
[ https://issues.apache.org/jira/browse/YARN-5946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883905#comment-15883905 ] Jonathan Hung commented on YARN-5946: - Thanks Xuan and Wangda! > Create YarnConfigurationStore interface and InMemoryConfigurationStore class > > > Key: YARN-5946 > URL: https://issues.apache.org/jira/browse/YARN-5946 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Fix For: YARN-5734 > > Attachments: YARN-5946.001.patch, YARN-5946-YARN-5734.002.patch, > YARN-5946-YARN-5734.003.patch, YARN-5946-YARN-5734.004.patch > > > This class provides the interface to persist YARN configurations in a backing > store. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6189) Improve application status log message when RM restarted when app is in NEW state
[
https://issues.apache.org/jira/browse/YARN-6189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883891#comment-15883891
]
Hadoop QA commented on YARN-6189:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 39m 13s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 32s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6189 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12854622/YARN-6189-v2.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 275c1c8d7b38 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / d2b3ba9 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/15082/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/15082/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15082/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Improve application status log message when RM restarted when app is in NEW
> state
> -
>
> Key: YARN-6189
>
[jira] [Commented] (YARN-6189) Improve application status log message when RM restarted when app is in NEW state
[
https://issues.apache.org/jira/browse/YARN-6189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883822#comment-15883822
]
Junping Du commented on YARN-6189:
--
Thanks [~templedf] for review. v2 patch should incorporate your comments (and
fix a compile issue).
> Improve application status log message when RM restarted when app is in NEW
> state
> -
>
> Key: YARN-6189
> URL: https://issues.apache.org/jira/browse/YARN-6189
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Yesha Vora
>Assignee: Junping Du
> Attachments: YARN-6189.patch, YARN-6189-v2.patch
>
>
> When RM restart/failover happens when application is in NEW state,
> application status command for that application prints below stacktrace.
> Improve exception message to less confusion to say something like:
> "application is not unknown, may be previous submission is
> not successful."
> {code}
> hrt_qa@:/root> yarn application -status application_1470379565464_0001
> 16/08/05 17:24:29 INFO impl.TimelineClientImpl: Timeline service address:
> https://hostxxx:8190/ws/v1/timeline/
> 16/08/05 17:24:30 INFO client.AHSProxy: Connecting to Application History
> server at hostxxx/xxx:10200
> 16/08/05 17:24:31 WARN retry.RetryInvocationHandler: Exception while invoking
> ApplicationClientProtocolPBClientImpl.getApplicationReport over rm1. Not
> retrying because try once and fail.
> org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application
> with id 'application_1470379565464_0001' doesn't exist in RM.
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:331)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:175)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:417)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:101)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:194)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
> at com.sun.proxy.$Proxy18.getApplicationReport(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:436)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.printApplicationReport(ApplicationCLI.java:481)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:160)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.main(ApplicationCLI.java:83)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException):
> Application with id 'application_1470379565464_0001' doesn't exist in RM.
>
[jira] [Updated] (YARN-6189) Improve application status log message when RM restarted when app is in NEW state
[
https://issues.apache.org/jira/browse/YARN-6189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated YARN-6189:
-
Attachment: YARN-6189-v2.patch
> Improve application status log message when RM restarted when app is in NEW
> state
> -
>
> Key: YARN-6189
> URL: https://issues.apache.org/jira/browse/YARN-6189
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Yesha Vora
>Assignee: Junping Du
> Attachments: YARN-6189.patch, YARN-6189-v2.patch
>
>
> When RM restart/failover happens when application is in NEW state,
> application status command for that application prints below stacktrace.
> Improve exception message to less confusion to say something like:
> "application is not unknown, may be previous submission is
> not successful."
> {code}
> hrt_qa@:/root> yarn application -status application_1470379565464_0001
> 16/08/05 17:24:29 INFO impl.TimelineClientImpl: Timeline service address:
> https://hostxxx:8190/ws/v1/timeline/
> 16/08/05 17:24:30 INFO client.AHSProxy: Connecting to Application History
> server at hostxxx/xxx:10200
> 16/08/05 17:24:31 WARN retry.RetryInvocationHandler: Exception while invoking
> ApplicationClientProtocolPBClientImpl.getApplicationReport over rm1. Not
> retrying because try once and fail.
> org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application
> with id 'application_1470379565464_0001' doesn't exist in RM.
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:331)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:175)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:417)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:101)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:194)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
> at com.sun.proxy.$Proxy18.getApplicationReport(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:436)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.printApplicationReport(ApplicationCLI.java:481)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:160)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.main(ApplicationCLI.java:83)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException):
> Application with id 'application_1470379565464_0001' doesn't exist in RM.
> at
>
[jira] [Commented] (YARN-5602) Utils for Federation State and Policy Store
[
https://issues.apache.org/jira/browse/YARN-5602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883817#comment-15883817
]
Subru Krishnan commented on YARN-5602:
--
[~giovanni.fumarola], thanks for the patch. I looked at it and have a few minor
comments:
* I think you missed adding a comment on why _Capability_ and _HeartBeat_ are
not used for comparison as suggested by [~jianhe].
* I don't think we should not have HikariCP dependency in the utils.
Shouldn't that be part of the specific implementation?
* I feel that the {{FederationStateStoreErrorCode}} should correspond to the
API, as now multiple invocation failures that return the same error code?
* Nit: There are few public methods which don't have Javadocs.
> Utils for Federation State and Policy Store
> ---
>
> Key: YARN-5602
> URL: https://issues.apache.org/jira/browse/YARN-5602
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Giovanni Matteo Fumarola
>Assignee: Giovanni Matteo Fumarola
> Labels: oct16-medium
> Attachments: YARN-5602-YARN-2915.v1.patch,
> YARN-5602-YARN-2915.v2.patch, YARN-5602-YARN-2915.v3.patch,
> YARN-5602-YARN-2915.v4.patch
>
>
> This JIRA tracks the creation of utils for Federation State and Policy Store
> such as Error Codes, Exceptions...
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-6030) Eliminate timelineServiceV2 boolean flag in TimelineClientImpl
[
https://issues.apache.org/jira/browse/YARN-6030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibo Chen resolved YARN-6030.
--
Resolution: Not A Problem
> Eliminate timelineServiceV2 boolean flag in TimelineClientImpl
> --
>
> Key: YARN-6030
> URL: https://issues.apache.org/jira/browse/YARN-6030
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-5355
>Reporter: Li Lu
>Priority: Minor
>
> I just discovered that we're still using a boolean flag {{timelineServiceV2}}
> after we introduced {{timelineServiceVersion}}. This sounds a little bit
> error-pruning. After the discussion I think we should only use and trust
> {{timelineServiceVersion}}. {{timelineServiceV2}} is set upon client
> creation. Instead of creating a v2 client and set this flag, maybe we'd like
> to do some sanity check and make sure the creation call is consistent with
> the configuration?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5946) Create YarnConfigurationStore interface and InMemoryConfigurationStore class
[ https://issues.apache.org/jira/browse/YARN-5946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883798#comment-15883798 ] Xuan Gong commented on YARN-5946: - Committed into YARN-5946. Thanks, Jonathan for working on this. And Thanks, wangda for review > Create YarnConfigurationStore interface and InMemoryConfigurationStore class > > > Key: YARN-5946 > URL: https://issues.apache.org/jira/browse/YARN-5946 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Fix For: YARN-5734 > > Attachments: YARN-5946.001.patch, YARN-5946-YARN-5734.002.patch, > YARN-5946-YARN-5734.003.patch, YARN-5946-YARN-5734.004.patch > > > This class provides the interface to persist YARN configurations in a backing > store. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6030) Eliminate timelineServiceV2 boolean flag in TimelineClientImpl
[
https://issues.apache.org/jira/browse/YARN-6030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883782#comment-15883782
]
Li Lu commented on YARN-6030:
-
I think so. Please feel free to check and close. Thanks!
> Eliminate timelineServiceV2 boolean flag in TimelineClientImpl
> --
>
> Key: YARN-6030
> URL: https://issues.apache.org/jira/browse/YARN-6030
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-5355
>Reporter: Li Lu
>Priority: Minor
>
> I just discovered that we're still using a boolean flag {{timelineServiceV2}}
> after we introduced {{timelineServiceVersion}}. This sounds a little bit
> error-pruning. After the discussion I think we should only use and trust
> {{timelineServiceVersion}}. {{timelineServiceV2}} is set upon client
> creation. Instead of creating a v2 client and set this flag, maybe we'd like
> to do some sanity check and make sure the creation call is consistent with
> the configuration?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5946) Create YarnConfigurationStore interface and InMemoryConfigurationStore class
[ https://issues.apache.org/jira/browse/YARN-5946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883784#comment-15883784 ] Xuan Gong commented on YARN-5946: - +1 Committing > Create YarnConfigurationStore interface and InMemoryConfigurationStore class > > > Key: YARN-5946 > URL: https://issues.apache.org/jira/browse/YARN-5946 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Fix For: YARN-5734 > > Attachments: YARN-5946.001.patch, YARN-5946-YARN-5734.002.patch, > YARN-5946-YARN-5734.003.patch, YARN-5946-YARN-5734.004.patch > > > This class provides the interface to persist YARN configurations in a backing > store. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6030) Eliminate timelineServiceV2 boolean flag in TimelineClientImpl
[
https://issues.apache.org/jira/browse/YARN-6030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883759#comment-15883759
]
Haibo Chen commented on YARN-6030:
--
This issue seems to be no longer valid after YARN-4675, [~gtCarrera9]?
> Eliminate timelineServiceV2 boolean flag in TimelineClientImpl
> --
>
> Key: YARN-6030
> URL: https://issues.apache.org/jira/browse/YARN-6030
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-5355
>Reporter: Li Lu
>Priority: Minor
>
> I just discovered that we're still using a boolean flag {{timelineServiceV2}}
> after we introduced {{timelineServiceVersion}}. This sounds a little bit
> error-pruning. After the discussion I think we should only use and trust
> {{timelineServiceVersion}}. {{timelineServiceV2}} is set upon client
> creation. Instead of creating a v2 client and set this flag, maybe we'd like
> to do some sanity check and make sure the creation call is consistent with
> the configuration?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5548) Use MockRMMemoryStateStore to reduce test failures
[
https://issues.apache.org/jira/browse/YARN-5548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883656#comment-15883656
]
Hadoop QA commented on YARN-5548:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} YARN-5548 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-5548 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12841536/YARN-5548.0010.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15081/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Use MockRMMemoryStateStore to reduce test failures
> --
>
> Key: YARN-5548
> URL: https://issues.apache.org/jira/browse/YARN-5548
> Project: Hadoop YARN
> Issue Type: Test
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Labels: oct16-easy, test
> Attachments: YARN-5548.0001.patch, YARN-5548.0002.patch,
> YARN-5548.0003.patch, YARN-5548.0004.patch, YARN-5548.0005.patch,
> YARN-5548.0006.patch, YARN-5548.0007.patch, YARN-5548.0008.patch,
> YARN-5548.0009.patch, YARN-5548.0010.patch
>
>
> https://builds.apache.org/job/PreCommit-YARN-Build/12850/testReport/org.apache.hadoop.yarn.server.resourcemanager/TestRMRestart/testFinishedAppRemovalAfterRMRestart/
> {noformat}
> Error Message
> Stacktrace
> java.lang.AssertionError: expected null, but was: application_submission_context { application_id { id: 1 cluster_timestamp:
> 1471885197388 } application_name: "" queue: "default" priority { priority: 0
> } am_container_spec { } cancel_tokens_when_complete: true maxAppAttempts: 2
> resource { memory: 1024 virtual_cores: 1 } applicationType: "YARN"
> keep_containers_across_application_attempts: false
> attempt_failures_validity_interval: 0 am_container_resource_request {
> priority { priority: 0 } resource_name: "*" capability { memory: 1024
> virtual_cores: 1 } num_containers: 0 relax_locality: true
> node_label_expression: "" execution_type_request { execution_type: GUARANTEED
> enforce_execution_type: false } } } user: "jenkins" start_time: 1471885197417
> application_state: RMAPP_FINISHED finish_time: 1471885197478>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotNull(Assert.java:664)
> at org.junit.Assert.assertNull(Assert.java:646)
> at org.junit.Assert.assertNull(Assert.java:656)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testFinishedAppRemovalAfterRMRestart(TestRMRestart.java:1656)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6235) YARN UI - expand the tool bars by default
Jian He created YARN-6235: - Summary: YARN UI - expand the tool bars by default Key: YARN-6235 URL: https://issues.apache.org/jira/browse/YARN-6235 Project: Hadoop YARN Issue Type: Bug Reporter: Jian He On RM UI, We have tool bars on the left hand side which is collapsed by default. Similarly, on the NodeManager UI, only the current list is expanded by default. Just because they are collapsed, some people not familiar with the UI don't know they are expandable. IMO, there are anyways much space left on the UI, why not expand all of them by default to be more clear for first-sight and also more convenient to navigate. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5548) Use MockRMMemoryStateStore to reduce test failures
[
https://issues.apache.org/jira/browse/YARN-5548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883645#comment-15883645
]
Miklos Szegedi commented on YARN-5548:
--
Thank you, [~bibinchundatt] for the patch. I see this pretty often recently. It
would be very helpful to get this fix checked in.
> Use MockRMMemoryStateStore to reduce test failures
> --
>
> Key: YARN-5548
> URL: https://issues.apache.org/jira/browse/YARN-5548
> Project: Hadoop YARN
> Issue Type: Test
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Labels: oct16-easy, test
> Attachments: YARN-5548.0001.patch, YARN-5548.0002.patch,
> YARN-5548.0003.patch, YARN-5548.0004.patch, YARN-5548.0005.patch,
> YARN-5548.0006.patch, YARN-5548.0007.patch, YARN-5548.0008.patch,
> YARN-5548.0009.patch, YARN-5548.0010.patch
>
>
> https://builds.apache.org/job/PreCommit-YARN-Build/12850/testReport/org.apache.hadoop.yarn.server.resourcemanager/TestRMRestart/testFinishedAppRemovalAfterRMRestart/
> {noformat}
> Error Message
> Stacktrace
> java.lang.AssertionError: expected null, but was: application_submission_context { application_id { id: 1 cluster_timestamp:
> 1471885197388 } application_name: "" queue: "default" priority { priority: 0
> } am_container_spec { } cancel_tokens_when_complete: true maxAppAttempts: 2
> resource { memory: 1024 virtual_cores: 1 } applicationType: "YARN"
> keep_containers_across_application_attempts: false
> attempt_failures_validity_interval: 0 am_container_resource_request {
> priority { priority: 0 } resource_name: "*" capability { memory: 1024
> virtual_cores: 1 } num_containers: 0 relax_locality: true
> node_label_expression: "" execution_type_request { execution_type: GUARANTEED
> enforce_execution_type: false } } } user: "jenkins" start_time: 1471885197417
> application_state: RMAPP_FINISHED finish_time: 1471885197478>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotNull(Assert.java:664)
> at org.junit.Assert.assertNull(Assert.java:646)
> at org.junit.Assert.assertNull(Assert.java:656)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testFinishedAppRemovalAfterRMRestart(TestRMRestart.java:1656)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6189) Improve application status log message when RM restarted when app is in NEW state
[
https://issues.apache.org/jira/browse/YARN-6189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883622#comment-15883622
]
Hadoop QA commented on YARN-6189:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
22s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
21s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 21s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 22s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
24s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
15s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 23s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6189 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12854579/YARN-6189.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 6cb580d87639 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 289bc50 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| mvninstall |
https://builds.apache.org/job/PreCommit-YARN-Build/15080/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| compile |
https://builds.apache.org/job/PreCommit-YARN-Build/15080/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| javac |
https://builds.apache.org/job/PreCommit-YARN-Build/15080/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| checkstyle |
[jira] [Commented] (YARN-5280) Allow YARN containers to run with Java Security Manager
[ https://issues.apache.org/jira/browse/YARN-5280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883618#comment-15883618 ] Robert Kanter commented on YARN-5280: - Will commit this later today if no other comments > Allow YARN containers to run with Java Security Manager > --- > > Key: YARN-5280 > URL: https://issues.apache.org/jira/browse/YARN-5280 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager, yarn >Affects Versions: 2.6.4 >Reporter: Greg Phillips >Assignee: Greg Phillips >Priority: Minor > Labels: oct16-medium > Attachments: YARN-5280.001.patch, YARN-5280.002.patch, > YARN-5280.003.patch, YARN-5280.004.patch, YARN-5280.005.patch, > YARN-5280.006.patch, YARN-5280.007.patch, YARN-5280.008.patch, > YARN-5280.patch, YARNContainerSandbox.pdf > > > YARN applications have the ability to perform privileged actions which have > the potential to add instability into the cluster. The Java Security Manager > can be used to prevent users from running privileged actions while still > allowing their core data processing use cases. > Introduce a YARN flag which will allow a Hadoop administrator to enable the > Java Security Manager for user code, while still providing complete > permissions to core Hadoop libraries. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6140) start time key in NM leveldb store should be removed when container is removed
[
https://issues.apache.org/jira/browse/YARN-6140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883609#comment-15883609
]
Sangjin Lee commented on YARN-6140:
---
Any update on this [~ajithshetty]?
> start time key in NM leveldb store should be removed when container is removed
> --
>
> Key: YARN-6140
> URL: https://issues.apache.org/jira/browse/YARN-6140
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Affects Versions: YARN-5355
>Reporter: Sangjin Lee
>Assignee: Ajith S
> Labels: yarn-5355-merge-blocker
>
> It appears that the start time key is not removed when the container is
> removed. The key was introduced in YARN-5792.
> I found this while backporting the YARN-5355-branch-2 branch to our internal
> branch loosely based on 2.6.0. The {{TestNMLeveldbStateStoreService}} test
> was failing because of this.
> I'm not sure why we didn't see this earlier.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6189) Improve application status log message when RM restarted when app is in NEW state
[
https://issues.apache.org/jira/browse/YARN-6189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883603#comment-15883603
]
Daniel Templeton commented on YARN-6189:
Thanks, [~djp] for the patch. Sorry I didn't chime in earlier. I think the
text would be more correct/readable as "Application with id
'application_1470379565464_0001' doesn't exist in RM. Please check that the
job submission was successful."
It would be even better if we could lose the stack trace.
> Improve application status log message when RM restarted when app is in NEW
> state
> -
>
> Key: YARN-6189
> URL: https://issues.apache.org/jira/browse/YARN-6189
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Yesha Vora
>Assignee: Junping Du
> Attachments: YARN-6189.patch
>
>
> When RM restart/failover happens when application is in NEW state,
> application status command for that application prints below stacktrace.
> Improve exception message to less confusion to say something like:
> "application is not unknown, may be previous submission is
> not successful."
> {code}
> hrt_qa@:/root> yarn application -status application_1470379565464_0001
> 16/08/05 17:24:29 INFO impl.TimelineClientImpl: Timeline service address:
> https://hostxxx:8190/ws/v1/timeline/
> 16/08/05 17:24:30 INFO client.AHSProxy: Connecting to Application History
> server at hostxxx/xxx:10200
> 16/08/05 17:24:31 WARN retry.RetryInvocationHandler: Exception while invoking
> ApplicationClientProtocolPBClientImpl.getApplicationReport over rm1. Not
> retrying because try once and fail.
> org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application
> with id 'application_1470379565464_0001' doesn't exist in RM.
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:331)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:175)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:417)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:101)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:194)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
> at com.sun.proxy.$Proxy18.getApplicationReport(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:436)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.printApplicationReport(ApplicationCLI.java:481)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:160)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
>
[jira] [Updated] (YARN-6189) Improve application status log message when RM restarted when app is in NEW state
[
https://issues.apache.org/jira/browse/YARN-6189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated YARN-6189:
-
Attachment: YARN-6189.patch
Thanks [~yeshavora] for reporting the issue. I agree that the message here
could be a bit confusing to end users. Upload a quick patch to fix this.
> Improve application status log message when RM restarted when app is in NEW
> state
> -
>
> Key: YARN-6189
> URL: https://issues.apache.org/jira/browse/YARN-6189
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Yesha Vora
>Assignee: Junping Du
> Attachments: YARN-6189.patch
>
>
> When RM restart/failover happens when application is in NEW state,
> application status command for that application prints below stacktrace.
> Improve exception message to less confusion to say something like:
> "application is not unknown, may be previous submission is
> not successful."
> {code}
> hrt_qa@:/root> yarn application -status application_1470379565464_0001
> 16/08/05 17:24:29 INFO impl.TimelineClientImpl: Timeline service address:
> https://hostxxx:8190/ws/v1/timeline/
> 16/08/05 17:24:30 INFO client.AHSProxy: Connecting to Application History
> server at hostxxx/xxx:10200
> 16/08/05 17:24:31 WARN retry.RetryInvocationHandler: Exception while invoking
> ApplicationClientProtocolPBClientImpl.getApplicationReport over rm1. Not
> retrying because try once and fail.
> org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application
> with id 'application_1470379565464_0001' doesn't exist in RM.
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:331)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:175)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:417)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:101)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:194)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
> at com.sun.proxy.$Proxy18.getApplicationReport(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:436)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.printApplicationReport(ApplicationCLI.java:481)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:160)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.main(ApplicationCLI.java:83)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException):
> Application with id 'application_1470379565464_0001'
[jira] [Commented] (YARN-5602) Utils for Federation State and Policy Store
[ https://issues.apache.org/jira/browse/YARN-5602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883561#comment-15883561 ] Jian He commented on YARN-5602: --- lgtm, thanks > Utils for Federation State and Policy Store > --- > > Key: YARN-5602 > URL: https://issues.apache.org/jira/browse/YARN-5602 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola > Labels: oct16-medium > Attachments: YARN-5602-YARN-2915.v1.patch, > YARN-5602-YARN-2915.v2.patch, YARN-5602-YARN-2915.v3.patch, > YARN-5602-YARN-2915.v4.patch > > > This JIRA tracks the creation of utils for Federation State and Policy Store > such as Error Codes, Exceptions... -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4113) RM should respect retry-interval when uses RetryPolicies.RETRY_FOREVER
[
https://issues.apache.org/jira/browse/YARN-4113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated YARN-4113:
--
Fix Version/s: 3.0.0-alpha1
2.8.0
Old JIRA missing fix-versions. Setting them.
> RM should respect retry-interval when uses RetryPolicies.RETRY_FOREVER
> --
>
> Key: YARN-4113
> URL: https://issues.apache.org/jira/browse/YARN-4113
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Wangda Tan
>Assignee: Sunil G
>Priority: Critical
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: 0001-YARN-4113.patch
>
>
> Found one issue in RMProxy how to initialize RetryPolicy: In
> RMProxy#createRetryPolicy. When rmConnectWaitMS is set to -1 (wait forever),
> it uses RetryPolicies.RETRY_FOREVER which doesn't respect
> {{yarn.resourcemanager.connect.retry-interval.ms}} setting.
> RetryPolicies.RETRY_FOREVER uses 0 as the interval, when I run the test
> without properly setup localhost name:
> {{TestYarnClient#testShouldNotRetryForeverForNonNetworkExceptions}}, it wrote
> 14G DEBUG exception message to system before it dies. This will be very bad
> if we do the same thing in a production cluster.
> We should fix two places:
> - Make RETRY_FOREVER can take retry-interval as constructor parameter.
> - Respect retry-interval when we uses RETRY_FOREVER policy.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6189) Improve application status log message when RM restarted when app is in NEW state
[
https://issues.apache.org/jira/browse/YARN-6189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du reassigned YARN-6189:
Assignee: Junping Du
> Improve application status log message when RM restarted when app is in NEW
> state
> -
>
> Key: YARN-6189
> URL: https://issues.apache.org/jira/browse/YARN-6189
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Yesha Vora
>Assignee: Junping Du
>
> When RM restart/failover happens when application is in NEW state,
> application status command for that application prints below stacktrace.
> Improve exception message to less confusion to say something like:
> "application is not unknown, may be previous submission is
> not successful."
> {code}
> hrt_qa@:/root> yarn application -status application_1470379565464_0001
> 16/08/05 17:24:29 INFO impl.TimelineClientImpl: Timeline service address:
> https://hostxxx:8190/ws/v1/timeline/
> 16/08/05 17:24:30 INFO client.AHSProxy: Connecting to Application History
> server at hostxxx/xxx:10200
> 16/08/05 17:24:31 WARN retry.RetryInvocationHandler: Exception while invoking
> ApplicationClientProtocolPBClientImpl.getApplicationReport over rm1. Not
> retrying because try once and fail.
> org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application
> with id 'application_1470379565464_0001' doesn't exist in RM.
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:331)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:175)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:417)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:101)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:194)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
> at com.sun.proxy.$Proxy18.getApplicationReport(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:436)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.printApplicationReport(ApplicationCLI.java:481)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:160)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.main(ApplicationCLI.java:83)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException):
> Application with id 'application_1470379565464_0001' doesn't exist in RM.
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:331)
> at
>
[jira] [Created] (YARN-6234) Support multiple attempts on the node when AMRMProxy is enabled
Subru Krishnan created YARN-6234:
Summary: Support multiple attempts on the node when AMRMProxy is
enabled
Key: YARN-6234
URL: https://issues.apache.org/jira/browse/YARN-6234
Project: Hadoop YARN
Issue Type: Sub-task
Components: amrmproxy, federation, nodemanager
Affects Versions: 3.0.0-alpha1, 2.8.0
Reporter: Subru Krishnan
Assignee: Giovanni Matteo Fumarola
Currently {{AMRMProxy}} initializes an interceptor chain pipeline for every
active AM in the node but it doesn't clean up & reinitialize correctly if
there's a second attempt for any AM in the same node. This jira is to track the
changes required to support multiple attempts on the node when AMRMProxy is
enabled.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6016) Fix minor bugs in handling of local AMRMToken in AMRMProxy
[ https://issues.apache.org/jira/browse/YARN-6016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-6016: - Issue Type: Sub-task (was: Bug) Parent: YARN-2915 > Fix minor bugs in handling of local AMRMToken in AMRMProxy > -- > > Key: YARN-6016 > URL: https://issues.apache.org/jira/browse/YARN-6016 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Minor > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-6016.v1.patch, YARN-6016.v2.patch, > YARN-6016.v3.patch > > > Two AMRMProxy bugs: > First, the AMRMToken from RM should not be propagated to AM, since AMRMProxy > will create a local AMRMToken for it. > Second, the AMRMProxy Context is now parse the localAMRMTokenKeyId from > amrmToken, but should be from localAmrmToken. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6027) Support fromid(offset) filter for /flows API
[ https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883397#comment-15883397 ] Sangjin Lee commented on YARN-6027: --- Thanks for the clarification. I forgot that this is something that would be part of URLs and thus needs a string representation. Yes, at least a separate interface might be desirable. Mixing them in with the RowKey classes might be a little confusing. > Support fromid(offset) filter for /flows API > > > Key: YARN-6027 > URL: https://issues.apache.org/jira/browse/YARN-6027 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Labels: yarn-5355-merge-blocker > Attachments: YARN-6027-YARN-5355.0001.patch, > YARN-6027-YARN-5355.0002.patch, YARN-6027-YARN-5355.0003.patch, > YARN-6027-YARN-5355.0004.patch > > > In YARN-5585 , fromId is supported for retrieving entities. We need similar > filter for flows/flowRun apps and flow run and flow as well. > Along with supporting fromId, this JIRA should also discuss following points > * Should we throw an exception for entities/entity retrieval if duplicates > found? > * TimelieEntity : > ** Should equals method also check for idPrefix? > ** Does idPrefix is part of identifiers? -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5410) Bootstrap Router server module
[ https://issues.apache.org/jira/browse/YARN-5410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-5410: - Summary: Bootstrap Router server module (was: Bootstrap Router module) > Bootstrap Router server module > -- > > Key: YARN-5410 > URL: https://issues.apache.org/jira/browse/YARN-5410 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Giovanni Matteo Fumarola > Attachments: YARN-5410-YARN-2915-v1.patch, > YARN-5410-YARN-2915-v2.patch, YARN-5410-YARN-2915-v3.patch, > YARN-5410-YARN-2915-v4.patch, YARN-5410-YARN-2915-v5.patch, > YARN-5410-YARN-2915-v6.patch > > > As detailed in the proposal in the umbrella JIRA, we are introducing a new > component that routes client request to appropriate ResourceManager(s). This > JIRA tracks the creation of a new sub-module for the Router. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5410) Bootstrap Router module
[ https://issues.apache.org/jira/browse/YARN-5410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883368#comment-15883368 ] Wangda Tan commented on YARN-5410: -- [~subru], make sense to me. > Bootstrap Router module > --- > > Key: YARN-5410 > URL: https://issues.apache.org/jira/browse/YARN-5410 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Giovanni Matteo Fumarola > Attachments: YARN-5410-YARN-2915-v1.patch, > YARN-5410-YARN-2915-v2.patch, YARN-5410-YARN-2915-v3.patch, > YARN-5410-YARN-2915-v4.patch, YARN-5410-YARN-2915-v5.patch, > YARN-5410-YARN-2915-v6.patch > > > As detailed in the proposal in the umbrella JIRA, we are introducing a new > component that routes client request to appropriate ResourceManager(s). This > JIRA tracks the creation of a new sub-module for the Router. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6228) EntityGroupFSTimelineStore should allow configurable cache stores.
[ https://issues.apache.org/jira/browse/YARN-6228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883353#comment-15883353 ] Hudson commented on YARN-6228: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #11302 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11302/]) YARN-6228: EntityGroupFSTimelineStore should allow configurable cache (xgong: rev 53d372a2550c970f3dd3c49738af3c1789ae589b) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/EntityCacheItem.java > EntityGroupFSTimelineStore should allow configurable cache stores. > --- > > Key: YARN-6228 > URL: https://issues.apache.org/jira/browse/YARN-6228 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Li Lu > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: YARN-6228-trunk.001.patch, YARN-6228-trunk.002.patch > > > We should allow users to config which cache store to use for > EntityGroupFSTimelineStore. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6228) EntityGroupFSTimelineStore should allow configurable cache stores.
[ https://issues.apache.org/jira/browse/YARN-6228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-6228: Fix Version/s: 3.0.0-alpha3 2.9.0 > EntityGroupFSTimelineStore should allow configurable cache stores. > --- > > Key: YARN-6228 > URL: https://issues.apache.org/jira/browse/YARN-6228 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Li Lu > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: YARN-6228-trunk.001.patch, YARN-6228-trunk.002.patch > > > We should allow users to config which cache store to use for > EntityGroupFSTimelineStore. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6228) EntityGroupFSTimelineStore should allow configurable cache stores.
[ https://issues.apache.org/jira/browse/YARN-6228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883332#comment-15883332 ] Xuan Gong commented on YARN-6228: - Committed into trunk/branch-2. Thanks, Li > EntityGroupFSTimelineStore should allow configurable cache stores. > --- > > Key: YARN-6228 > URL: https://issues.apache.org/jira/browse/YARN-6228 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Li Lu > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: YARN-6228-trunk.001.patch, YARN-6228-trunk.002.patch > > > We should allow users to config which cache store to use for > EntityGroupFSTimelineStore. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6153) keepContainer does not work when AM retry window is set
[
https://issues.apache.org/jira/browse/YARN-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883323#comment-15883323
]
Jian He commented on YARN-6153:
---
[~kyungwan nam], thanks for updating, patch looks good to me overall,
I found there are several places in RMAppAttemptImpl where it uses below way to
retrieve its RMApp,
{code}
appAttempt.rmContext.getRMApps().get(
appAttempt.getAppAttemptId().getApplicationId()
{code}
I think we can change the RMAppAttemptImpl constructor to take RMApp as one
parameter so that we won't need the hashmap to back trace its RMApp, would you
like to make the change ?
> keepContainer does not work when AM retry window is set
> ---
>
> Key: YARN-6153
> URL: https://issues.apache.org/jira/browse/YARN-6153
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.7.1
>Reporter: kyungwan nam
> Attachments: YARN-6153.001.patch, YARN-6153.002.patch,
> YARN-6153.003.patch, YARN-6153.004.patch, YARN-6153.005.patch
>
>
> yarn.resourcemanager.am.max-attempts has been configured to 2 in my cluster.
> I submitted a YARN application (slider app) that keepContainers=true,
> attemptFailuresValidityInterval=30.
> it did work properly when AM was failed firstly.
> all containers launched by previous AM were resynced with new AM (attempt2)
> without killing containers.
> after 10 minutes, I thought AM failure count was reset by
> attemptFailuresValidityInterval (5 minutes).
> but, all containers were killed when AM was failed secondly. (new AM attempt3
> was launched properly)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1728) History server doesn't understand percent encoded paths
[
https://issues.apache.org/jira/browse/YARN-1728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883241#comment-15883241
]
Gera Shegalov commented on YARN-1728:
-
Hi [~yuanbo], thanks for addressing the issue. I see that Guice itself [fixed
it|https://github.com/google/guice/pull/860/files] using
{{java.net.URI#getPath}}. Let us use it here so the behavior is consistent with
newer Guice.
I suggest we use:
{code}
decodedPathInfo = URI.create(pathInfo).getPath();
{code}
> History server doesn't understand percent encoded paths
> ---
>
> Key: YARN-1728
> URL: https://issues.apache.org/jira/browse/YARN-1728
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Abraham Elmahrek
>Assignee: Yuanbo Liu
> Attachments: YARN-1728-branch-2.001.patch,
> YARN-1728-branch-2.002.patch, YARN-1728-branch-2.003.patch
>
>
> For example, going to the job history server page
> http://localhost:19888/jobhistory/logs/localhost%3A8041/container_1391466602060_0011_01_01/job_1391466602060_0011/admin/stderr
> results in the following error:
> {code}
> Cannot get container logs. Invalid nodeId:
> test-cdh5-hue.ent.cloudera.com%3A8041
> {code}
> Where the url decoded version works:
> http://localhost:19888/jobhistory/logs/localhost:8041/container_1391466602060_0011_01_01/job_1391466602060_0011/admin/stderr
> It seems like both should be supported as the former is simply percent
> encoding.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6123) [YARN-5864] Add a test to make sure queues of orderingPolicy will be updated when childQueues is added or removed.
[
https://issues.apache.org/jira/browse/YARN-6123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883230#comment-15883230
]
Hadoop QA commented on YARN-6123:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} YARN-6123 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-6123 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12849549/YARN-6123.002.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15079/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> [YARN-5864] Add a test to make sure queues of orderingPolicy will be updated
> when childQueues is added or removed.
> --
>
> Key: YARN-6123
> URL: https://issues.apache.org/jira/browse/YARN-6123
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-6123.001.patch, YARN-6123.002.patch
>
>
> YARN-5864 added queue ordering policy to ParentQueue, we need to make sure
> queues of QueueOrderingPolicy will be updated when any changes made for child
> queues.
> We need to add a test to make sure it works.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6042) Dump scheduler and queue state information into FairScheduler DEBUG log
[ https://issues.apache.org/jira/browse/YARN-6042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883221#comment-15883221 ] Ray Chiang commented on YARN-6042: -- After an offline conversation with [~templedf], it looks like it would be a good idea to put these queue updates pointing at a separate log4j property and make the default a separate file. That should help with the huge volume of information this debug setting will put in a log file. > Dump scheduler and queue state information into FairScheduler DEBUG log > --- > > Key: YARN-6042 > URL: https://issues.apache.org/jira/browse/YARN-6042 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Reporter: Yufei Gu >Assignee: Yufei Gu > Attachments: YARN-6042.001.patch, YARN-6042.002.patch, > YARN-6042.003.patch, YARN-6042.004.patch, YARN-6042.005.patch, > YARN-6042.006.patch, YARN-6042.007.patch > > > To improve the debugging of scheduler issues it would be a big improvement to > be able to dump the scheduler state into a log on request. > The Dump the scheduler state at a point in time would allow debugging of a > scheduler that is not hung (deadlocked) but also not assigning containers. > Currently we do not have a proper overview of what state the scheduler and > the queues are in and we have to make assumptions or guess > The scheduler and queue state needed would include (not exhaustive): > - instantaneous and steady fair share (app / queue) > - AM share and resources > - weight > - app demand > - application run state (runnable/non runnable) > - last time at fair/min share -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5517) Add GPU as a resource type for scheduling
[ https://issues.apache.org/jira/browse/YARN-5517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883209#comment-15883209 ] Daniel Templeton commented on YARN-5517: Yep, I agree. > Add GPU as a resource type for scheduling > - > > Key: YARN-5517 > URL: https://issues.apache.org/jira/browse/YARN-5517 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Jaeboo Jeong > Attachments: aggregate_resource_allocation.jpg, > container_example.jpg, RM-scheduler_metrics.jpg, YARN-5517-branch-2.7.1.patch > > > Currently YARN only support scheduling based on memory and cpu. > There is the issue(YARN-3926) which proposed to extend the YARN resource > model. > And there is the issue(YARN-4122) to add support for GPU as a resource using > docker. > But these issues didn’t release yet so I just added GPU resource type like > memory and cpu. > I don’t consider GPU isolation like YARN-4122. > The properties for GPU resource type is similar to cpu core. > mapred-default.xml > mapreduce.map.gpu.cores (default 0) > mapreduce.reduce.gpu.cores(default 0) > yarn.app.mapreduce.am.resource.gpu-cores (default 0) > yarn-default.xml > yarn.scheduler.minimum-allocation-gcores (default 0) > yarn.scheduler.maximum-allocation-gcores (default 8) > yarn.nodemanager.resource.gcores (default 0) > I attached the patch for branch-2.7.1 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6218) TestAMRMClient fails with fair scheduler
[ https://issues.apache.org/jira/browse/YARN-6218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883194#comment-15883194 ] Haibo Chen commented on YARN-6218: -- Thanks for the update, [~miklos.szeg...@cloudera.com]. One nit, the sleep(100) right after nmClient.getContainerStatus() is now changed to sleep(10). We should probably leave it as is. > TestAMRMClient fails with fair scheduler > > > Key: YARN-6218 > URL: https://issues.apache.org/jira/browse/YARN-6218 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Minor > Attachments: YARN-6218.000.patch, YARN-6218.001.patch > > > We ran into this issue on v2. Allocation does not happen in the specified > amount of time. > Error Message > expected:<2> but was:<0> > Stacktrace > java.lang.AssertionError: expected:<2> but was:<0> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at org.junit.Assert.assertEquals(Assert.java:555) > at org.junit.Assert.assertEquals(Assert.java:542) > at > org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.testAMRMClientMatchStorage(TestAMRMClient.java:495) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5517) Add GPU as a resource type for scheduling
[ https://issues.apache.org/jira/browse/YARN-5517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883192#comment-15883192 ] Wangda Tan commented on YARN-5517: -- Yeah, I agree to close this as dup of YARN-3926 . > Add GPU as a resource type for scheduling > - > > Key: YARN-5517 > URL: https://issues.apache.org/jira/browse/YARN-5517 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Jaeboo Jeong > Attachments: aggregate_resource_allocation.jpg, > container_example.jpg, RM-scheduler_metrics.jpg, YARN-5517-branch-2.7.1.patch > > > Currently YARN only support scheduling based on memory and cpu. > There is the issue(YARN-3926) which proposed to extend the YARN resource > model. > And there is the issue(YARN-4122) to add support for GPU as a resource using > docker. > But these issues didn’t release yet so I just added GPU resource type like > memory and cpu. > I don’t consider GPU isolation like YARN-4122. > The properties for GPU resource type is similar to cpu core. > mapred-default.xml > mapreduce.map.gpu.cores (default 0) > mapreduce.reduce.gpu.cores(default 0) > yarn.app.mapreduce.am.resource.gpu-cores (default 0) > yarn-default.xml > yarn.scheduler.minimum-allocation-gcores (default 0) > yarn.scheduler.maximum-allocation-gcores (default 8) > yarn.nodemanager.resource.gcores (default 0) > I attached the patch for branch-2.7.1 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4122) Add support for GPU as a resource
[ https://issues.apache.org/jira/browse/YARN-4122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883183#comment-15883183 ] Wangda Tan commented on YARN-4122: -- [~hex108], I would suggest to move this to sub ticket of YARN-6223 to support GPU isolation by using CGroups. Does it make sense to you? > Add support for GPU as a resource > - > > Key: YARN-4122 > URL: https://issues.apache.org/jira/browse/YARN-4122 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Jun Gong >Assignee: Jun Gong > Attachments: GPUAsAResourceDesign.pdf > > > Use [cgroups > devcies|https://www.kernel.org/doc/Documentation/cgroups/devices.txt] to > isolate GPUs for containers. For docker containers, we could use 'docker run > --device=...'. > Reference: [SLURM Resources isolation through > cgroups|http://slurm.schedmd.com/slurm_ug_2011/SLURM_UserGroup2011_cgroups.pdf]. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4779) Fix AM container allocation logic in SLS
[ https://issues.apache.org/jira/browse/YARN-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883178#comment-15883178 ] Wangda Tan commented on YARN-4779: -- Thanks [~sunilg] and [~jianhe] for review! bq. do we need to backport to branch-2 or branch-2.8 ? I believe there're some dependencies haven't been backported to branch-2, so let's keep it in trunk for now. > Fix AM container allocation logic in SLS > > > Key: YARN-4779 > URL: https://issues.apache.org/jira/browse/YARN-4779 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler-load-simulator >Reporter: Wangda Tan >Assignee: Wangda Tan > Labels: oct16-medium > Fix For: 3.0.0-alpha3 > > Attachments: YARN-4779.1.patch, YARN-4779.2.patch, YARN-4779.3.patch, > YARN-4779.4.patch, YARN-4779.5.patch > > > Currently, SLS uses unmanaged AM for simulated map-reduce applications. And > first allocated container for each app is considered to be the master > container. > This could be problematic when preemption happens. CapacityScheduler preempt > AM container at lowest priority, but the simulated AM container isn't > recognized by scheduler -- it is a normal container from scheduler's > perspective. > This JIRA tries to fix this logic: do the real AM allocation instead of using > unmanaged AM. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6233) FSRMStateStore UT fails with IO Timed out Error
Yesha Vora created YARN-6233:
Summary: FSRMStateStore UT fails with IO Timed out Error
Key: YARN-6233
URL: https://issues.apache.org/jira/browse/YARN-6233
Project: Hadoop YARN
Issue Type: Bug
Reporter: Yesha Vora
FSRMStateStore UT fails with IO Timed out Error as below.
{code:title=test cmd}
export MAVEN_OPTS=-Xmx1024m; mvn -B -nsu test
-Dtest=TestFifoScheduler,TestFairOrderingPolicy,TestFSAppAttempt,TestFSParentQueue,TestQueueManager,TestFairSchedulerFairShare,TestMaxRunningAppsEnforcer,TestAppRunnability,TestFairSchedulerConfiguration,TestFairSchedulerPreemption,TestSchedulingPolicy,TestComputeFairShares,TestFSLeafQueue,TestFairSchedulerEventLog,TestQueuePlacementPolicy,TestFairSchedulerQueueACLs,TestAllocationFileLoaderService,TestFairScheduler,TestDominantResourceFairnessPolicy,TestEmptyQueues,TestQueueCapacities,TestChildQueueOrder,TestQueueMappings,TestParentQueue,TestCapacitySchedulerNodeLabelUpdate,TestNodeLabelContainerAllocation,TestCapacityScheduler,TestApplicationLimits,TestWorkPreservingRMRestartForNodeLabel,TestReservationQueue,TestApplicationLimitsByPartition,TestCapacitySchedulerDynamicBehavior,TestQueueParsing,TestCapacitySchedulerLazyPreemption,TestContainerAllocation,TestLeafQueue,TestCapacitySchedulerSurgicalPreemption,TestReservations,TestCapacitySchedulerQueueACLs,TestUtils,TestPriorityUtilizationQueueOrderingPolicy,TestRMApplicationHistoryWriter,TestResources,TestResourceWeights,TestRMNMRPCResponseId,TestNMReconnect,TestNMExpiry,TestLeveldbRMStateStore,TestZKRMStateStore,TestMemoryRMStateStore,TestFSRMStateStore,TestZKRMStateStoreZKClientConnections,TestSystemMetricsPublisher,TestSimpleCapacityReplanner,TestInMemoryPlan,TestNoOverCommitPolicy,TestRLESparseResourceAllocation,TestCapacitySchedulerPlanFollower,TestInMemoryReservationAllocation,TestSchedulerPlanFollowerBase,TestGreedyReservationAgent,TestReservationInputValidator,TestRpcCall
--projects :hadoop-yarn-server-resourcemanager,:hadoop-nfs{code}
{code}
Results :
Tests in error:
TestFSRMStateStore.testFSRMStateStoreClientRetry:385 » test timed out after
3...
TestFSRMStateStore.testFSRMStateStore:168 » IO Timed out waiting for Mini
HDFS...
Tests run: 487, Failures: 0, Errors: 2, Skipped: 2
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop NFS .. SUCCESS [ 4.172 s]
[INFO] hadoop-yarn-server-resourcemanager . FAILURE [21:57 min]
[INFO]
[INFO] BUILD FAILURE
[INFO]
[INFO] Total time: 22:05 min
[INFO] Finished at: 2017-02-23T21:33:03+00:00
[INFO] Final Memory: 53M/873M
[INFO]
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on
project hadoop-yarn-server-resourcemanager: There are test failures.
[ERROR]
[ERROR] Please refer to
/xxx/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/target/surefire-reports
for the individual test results.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hadoop-yarn-server-resourcemanager{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1728) History server doesn't understand percent encoded paths
[
https://issues.apache.org/jira/browse/YARN-1728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883115#comment-15883115
]
Haibo Chen commented on YARN-1728:
--
Latest patch looks good to me. non-binding +1
> History server doesn't understand percent encoded paths
> ---
>
> Key: YARN-1728
> URL: https://issues.apache.org/jira/browse/YARN-1728
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Abraham Elmahrek
>Assignee: Yuanbo Liu
> Attachments: YARN-1728-branch-2.001.patch,
> YARN-1728-branch-2.002.patch, YARN-1728-branch-2.003.patch
>
>
> For example, going to the job history server page
> http://localhost:19888/jobhistory/logs/localhost%3A8041/container_1391466602060_0011_01_01/job_1391466602060_0011/admin/stderr
> results in the following error:
> {code}
> Cannot get container logs. Invalid nodeId:
> test-cdh5-hue.ent.cloudera.com%3A8041
> {code}
> Where the url decoded version works:
> http://localhost:19888/jobhistory/logs/localhost:8041/container_1391466602060_0011_01_01/job_1391466602060_0011/admin/stderr
> It seems like both should be supported as the former is simply percent
> encoding.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4779) Fix AM container allocation logic in SLS
[ https://issues.apache.org/jira/browse/YARN-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883033#comment-15883033 ] Hudson commented on YARN-4779: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11301 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11301/]) YARN-4779. Fix AM container allocation logic in SLS. Contributed by (sunilg: rev b32ffa2753e83615b980721b6067fcc35ce54372) * (edit) hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/MRAMSimulator.java * (edit) hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/AMSimulator.java * (edit) hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java * (add) hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/resourcemanager/MockAMLauncher.java * (edit) hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java > Fix AM container allocation logic in SLS > > > Key: YARN-4779 > URL: https://issues.apache.org/jira/browse/YARN-4779 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler-load-simulator >Reporter: Wangda Tan >Assignee: Wangda Tan > Labels: oct16-medium > Fix For: 3.0.0-alpha3 > > Attachments: YARN-4779.1.patch, YARN-4779.2.patch, YARN-4779.3.patch, > YARN-4779.4.patch, YARN-4779.5.patch > > > Currently, SLS uses unmanaged AM for simulated map-reduce applications. And > first allocated container for each app is considered to be the master > container. > This could be problematic when preemption happens. CapacityScheduler preempt > AM container at lowest priority, but the simulated AM container isn't > recognized by scheduler -- it is a normal container from scheduler's > perspective. > This JIRA tries to fix this logic: do the real AM allocation instead of using > unmanaged AM. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5892) Capacity Scheduler: Support user-specific minimum user limit percent
[
https://issues.apache.org/jira/browse/YARN-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882720#comment-15882720
]
Eric Payne commented on YARN-5892:
--
Thanks, [~leftnoteasy], for your feedback. I really value your input.
{quote}
in my mind there're some alternative solutions:
a. Create queue just for such vip users
{quote}
In our multi-tenant clusters, we have several users (sometimes dozens) needing
to use the same queue. Setting up separate queues for each of them based on
weighted importance is more complicated than giving each users their own weight.
bq. #1, if there're N (N <= 100 / MULP) users are consuming resource in a
queue, each of them can get at least MULP / 100 * queue-configured-capacity.
Even today, we can have N > 100/MULP. If I think of these _VIP_ users being the
weighted as multiple users, then we have a similar situation. In your example
above, Jack and Alice would be weighted as 1 user, but Admin would be 2.5 users.
> Capacity Scheduler: Support user-specific minimum user limit percent
>
>
> Key: YARN-5892
> URL: https://issues.apache.org/jira/browse/YARN-5892
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
>Reporter: Eric Payne
>Assignee: Eric Payne
> Attachments: YARN-5892.001.patch, YARN-5892.002.patch
>
>
> Currently, in the capacity scheduler, the {{minimum-user-limit-percent}}
> property is per queue. A cluster admin should be able to set the minimum user
> limit percent on a per-user basis within the queue.
> This functionality is needed so that when intra-queue preemption is enabled
> (YARN-4945 / YARN-2113), some users can be deemed as more important than
> other users, and resources from VIP users won't be as likely to be preempted.
> For example, if the {{getstuffdone}} queue has a MULP of 25 percent, but user
> {{jane}} is a power user of queue {{getstuffdone}} and needs to be guaranteed
> 75 percent, the properties for {{getstuffdone}} and {{jane}} would look like
> this:
> {code}
>
>
> yarn.scheduler.capacity.root.getstuffdone.minimum-user-limit-percent
> 25
>
>
>
> yarn.scheduler.capacity.root.getstuffdone.jane.minimum-user-limit-percent
> 75
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5588) Add support for resource profiles in distributed shell
[ https://issues.apache.org/jira/browse/YARN-5588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882716#comment-15882716 ] Sunil G commented on YARN-5588: --- Thanks [~vvasudev]. +1. Patch generally looks fine for me. I could commit this if there are no other objections tomorrow. > Add support for resource profiles in distributed shell > -- > > Key: YARN-5588 > URL: https://issues.apache.org/jira/browse/YARN-5588 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: YARN-5588-YARN-3926.001.patch, > YARN-5588-YARN-3926.002.patch, YARN-5588-YARN-3926.003.patch, > YARN-5588-YARN-3926.004.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5335) Use em-table in app/nodes pages for new YARN UI
[ https://issues.apache.org/jira/browse/YARN-5335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882651#comment-15882651 ] Sunil G commented on YARN-5335: --- Thanks [~akhilpb]. [~leftnoteasy] , could you please take a look also. > Use em-table in app/nodes pages for new YARN UI > --- > > Key: YARN-5335 > URL: https://issues.apache.org/jira/browse/YARN-5335 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-5335.0001.patch, YARN-5335.0002.patch > > > Convert to em-table for better flexibility in nodes and app pages. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5151) [YARN-3368] Support kill application from new YARN UI
[ https://issues.apache.org/jira/browse/YARN-5151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882597#comment-15882597 ] Gergely Novák commented on YARN-5151: - This is just an initial patch. Some TODOs: - Confirmation dialog - Better refresh - Ajax error handling > [YARN-3368] Support kill application from new YARN UI > - > > Key: YARN-5151 > URL: https://issues.apache.org/jira/browse/YARN-5151 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Gergely Novák > Attachments: YARN-5151.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5151) [YARN-3368] Support kill application from new YARN UI
[ https://issues.apache.org/jira/browse/YARN-5151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Novák updated YARN-5151: Attachment: YARN-5151.001.patch > [YARN-3368] Support kill application from new YARN UI > - > > Key: YARN-5151 > URL: https://issues.apache.org/jira/browse/YARN-5151 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Gergely Novák > Attachments: YARN-5151.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5151) [YARN-3368] Support kill application from new YARN UI
[ https://issues.apache.org/jira/browse/YARN-5151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Novák reassigned YARN-5151: --- Assignee: Gergely Novák (was: Sunil G) > [YARN-3368] Support kill application from new YARN UI > - > > Key: YARN-5151 > URL: https://issues.apache.org/jira/browse/YARN-5151 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Gergely Novák > Attachments: YARN-5151.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5588) Add support for resource profiles in distributed shell
[
https://issues.apache.org/jira/browse/YARN-5588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882571#comment-15882571
]
Hadoop QA commented on YARN-5588:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
7s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
4s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
28s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
25s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
6s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} YARN-3926 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
56s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 53s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 5 new + 303 unchanged - 13 fixed = 308 total (was 316) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
36s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 42m 24s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 45s{color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
22s{color} | {color:green} hadoop-yarn-applications-distributedshell in the
patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}134m 13s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMRestart |
| | hadoop.yarn.client.api.impl.TestAMRMProxy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5588 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12854431/YARN-5588-YARN-3926.004.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 1511a3971e1f 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-3926 / 7969a58 |
| Default Java |
[jira] [Commented] (YARN-6232) Update resource usage and preempted resource calculations to take into account all resource types
[
https://issues.apache.org/jira/browse/YARN-6232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882533#comment-15882533
]
Hadoop QA commented on YARN-6232:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 9 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
55s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
25s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
23s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
10s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
40s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
19s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
25s{color} | {color:green} YARN-3926 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 7m
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 7m 31s{color}
| {color:red} hadoop-yarn-project_hadoop-yarn generated 3 new + 38 unchanged -
0 fixed = 41 total (was 38) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 3s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 37 new + 729 unchanged - 19 fixed = 766 total (was 748) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
36s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 55s{color}
| {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
42s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 43m 42s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 17m 35s{color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}144m 26s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.api.TestPBImplRecords |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServiceAppsNodelabel |
| |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification |
| | hadoop.yarn.server.resourcemanager.TestContainerResourceUsage |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation |
| |
[jira] [Commented] (YARN-2487) Need to support timeout of AM When no containers are assigned to it for a defined period
[ https://issues.apache.org/jira/browse/YARN-2487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882439#comment-15882439 ] Sachin Aggarwal commented on YARN-2487: --- [~naganarasimha...@apache.org] [~rohithsharma] [~nijel], [~hejian991] [~wangda], I have a usecase which is similar to this. is there a chance you guys can consider this. or let me know if you have any work around this problem. I am running Jupyter kernel gateway in my custer. when JKG receives a request it starts a kernel in yarn-client mode. in yarn-client mode application master and executer runs in yarn and driver runs outside. In this case notebook comes up and kernel is running but I am not getting any container for master when other guys are using all resources. my aim here is. in such scenario I should wait for some time for resources and then I have to let user know that ur notebook is not able to get resources. please try after sometime and I kill his request. let know if you guys have more questions > Need to support timeout of AM When no containers are assigned to it for a > defined period > > > Key: YARN-2487 > URL: https://issues.apache.org/jira/browse/YARN-2487 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > > There are some scenarios where AM will not get containers and indefinitely > waiting. We faced one such sceanrio which makes the applications to get hung > : > Consider a cluster setup which has 2 NMS of each 8GB resource, > And 2 applications(MR2) are launched in the default queue where in each AM is > taking 2 GB each. > Each AM is placed in each of the NM. Now each AM is requesting for container > of 7Gb mem resource . > As in each NM only 6GB resource is available both the applications are hung > forever. > To avoid such scenarios i would like to propose > generic timeout feature for all AM's in yarn, such that if no containers are > assigned for an application for a defined period than yarn can timeout the > application attempt. > Default can be set to 0 where in RM will not timeout the app attempt and user > can set his own timeout when he submits the application -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5588) Add support for resource profiles in distributed shell
[
https://issues.apache.org/jira/browse/YARN-5588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Vasudev updated YARN-5588:
Attachment: YARN-5588-YARN-3926.004.patch
Thanks for the review [~sunilg].
bq. 1.1) Instead of having a variable named Resource none, we could use
Resource.NONE itself
The Resources class is part of hadoop-yarn-common but the ProfileCapability
class if part of hadoop-yarn-api. hadoop-yarn-common depends on hadoop-yarn-api.
bq. 1.2) tmp --> profileName
Fixed.
bq. 2.1) containerResourceProfile="", could we define a static variable as
EMPTY_PROFILE and could use every where.
I prefer not to do that for now - I suspect we will have to come back to handle
the empty profile and I'd like to leave that for later.
{quote}
2.2) In {[createProfileCapability}}, need to use containerMemory in below code
instead of containerVirtualCores
containerMemory = containerMemory == -1 ? DEFAULT_CONTAINER_MEMORY :
containerVirtualCores;
{quote}
Good catch. Fixed.
bq. 2.3) May be tmp could be renamed to profileName
Fixed.
bq. 3.1) for (String profile : appProfiles, could we rename profile ->
appProfile to avoid naming confusion.
Fixed.
bq. 3.2) In setAMResourceCapability, can amMemory from
profiles.get(tmp).getMemorySize() could be less than memory and cause Xmx issue.
The way the code works is that we only use profiles.get(tmp).getMemorySize()
when the command line parameter is not specified. So the scenario you specified
should not happen.
> Add support for resource profiles in distributed shell
> --
>
> Key: YARN-5588
> URL: https://issues.apache.org/jira/browse/YARN-5588
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: YARN-5588-YARN-3926.001.patch,
> YARN-5588-YARN-3926.002.patch, YARN-5588-YARN-3926.003.patch,
> YARN-5588-YARN-3926.004.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6153) keepContainer does not work when AM retry window is set
[
https://issues.apache.org/jira/browse/YARN-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882357#comment-15882357
]
Hadoop QA commented on YARN-6153:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
42s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 38s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 2m
5s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}103m 18s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.TestResourceTrackerService |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler |
| Timed out junit tests |
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStorePerf |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6153 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12854404/YARN-6153.005.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux e04da3151329 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 132f758 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/15075/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/15075/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15075/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> keepContainer does not work when AM
[jira] [Commented] (YARN-6207) Move application can fail when attempt add event is delayed
[
https://issues.apache.org/jira/browse/YARN-6207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882347#comment-15882347
]
Hadoop QA commented on YARN-6207:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 39m 26s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 39s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler |
| | hadoop.yarn.server.resourcemanager.TestRMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6207 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12854409/YARN-6207.005.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 671f58b5720d 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 132f758 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/15076/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/15076/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15076/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Move application can fail when attempt add event is delayed
>
>
> Key:
[jira] [Updated] (YARN-6232) Update resource usage and preempted resource calculations to take into account all resource types
[ https://issues.apache.org/jira/browse/YARN-6232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-6232: Component/s: (was: nodemanager) > Update resource usage and preempted resource calculations to take into > account all resource types > - > > Key: YARN-6232 > URL: https://issues.apache.org/jira/browse/YARN-6232 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > > The chargeback calculations that take place on the RM should be updated to > take all resource types into account. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6232) Update resource usage and preempted resource calculations to take into account all resource types
[ https://issues.apache.org/jira/browse/YARN-6232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-6232: Attachment: YARN-6232-YARN-3926.001.patch > Update resource usage and preempted resource calculations to take into > account all resource types > - > > Key: YARN-6232 > URL: https://issues.apache.org/jira/browse/YARN-6232 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: YARN-6232-YARN-3926.001.patch > > > The chargeback calculations that take place on the RM should be updated to > take all resource types into account. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6232) Update resource usage and preempted resource calculations to take into account all resource types
[ https://issues.apache.org/jira/browse/YARN-6232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-6232: Description: The chargeback calculations that take place on the RM should be updated to take all resource types into account. > Update resource usage and preempted resource calculations to take into > account all resource types > - > > Key: YARN-6232 > URL: https://issues.apache.org/jira/browse/YARN-6232 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > > The chargeback calculations that take place on the RM should be updated to > take all resource types into account. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6232) Update resource usage and preempted resource calculations to take into account all resource types
Varun Vasudev created YARN-6232: --- Summary: Update resource usage and preempted resource calculations to take into account all resource types Key: YARN-6232 URL: https://issues.apache.org/jira/browse/YARN-6232 Project: Hadoop YARN Issue Type: Sub-task Reporter: Varun Vasudev Assignee: Varun Vasudev -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6207) Move application can fail when attempt add event is delayed
[
https://issues.apache.org/jira/browse/YARN-6207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-6207:
---
Attachment: YARN-6207.005.patch
Attaching patch after handling comments and adding test cases
> Move application can fail when attempt add event is delayed
>
>
> Key: YARN-6207
> URL: https://issues.apache.org/jira/browse/YARN-6207
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: YARN-6207.001.patch, YARN-6207.002.patch,
> YARN-6207.003.patch, YARN-6207.004.patch, YARN-6207.005.patch
>
>
> *Steps to reproduce*
> 1.Submit application and delay attempt add to Scheduler
> (Simulate using debug at EventDispatcher for SchedulerEventDispatcher)
> 2. Call move application to destination queue.
> {noformat}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.preValidateMoveApplication(CapacityScheduler.java:2086)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.moveApplicationAcrossQueue(RMAppManager.java:669)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.moveApplicationAcrossQueues(ClientRMService.java:1231)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.moveApplicationAcrossQueues(ApplicationClientProtocolPBServiceImpl.java:388)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:537)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:522)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:867)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:813)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2659)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1483)
> at org.apache.hadoop.ipc.Client.call(Client.java:1429)
> at org.apache.hadoop.ipc.Client.call(Client.java:1339)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:115)
> at com.sun.proxy.$Proxy7.moveApplicationAcrossQueues(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.moveApplicationAcrossQueues(ApplicationClientProtocolPBClientImpl.java:398)
> ... 16 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6229) resource manager web UI display BUG
[ https://issues.apache.org/jira/browse/YARN-6229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] gehaijiang updated YARN-6229: - Attachment: rs.png attachment > resource manager web UI display BUG > - > > Key: YARN-6229 > URL: https://issues.apache.org/jira/browse/YARN-6229 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.1 > Environment: hadoop 2.7.1 >Reporter: gehaijiang > Attachments: rs.png > > > resourcemanager web UI display bug： > Memory Used -3.44TB > Containers Running -2607 > VCores Used -2607 > Lost Nodes 173 > These numbers are not correct。 > Cluster Metrics > Apps Submitted | Apps Pending | Apps Running |Apps Completed | > Containers Running | Memory Used | Memory Total | Memory Reserved |VCores > Used | VCores Total | VCores Reserved | Active Nodes | Decommissioned > Nodes | Lost Nodes | Unhealthy Nodes | Rebooted Nodes > 3027432 0 20 3027412 -2607 -3.44TB 9.70TB 0B -2607 > 72400 181 0 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6207) Move application can fail when attempt add event is delayed
[
https://issues.apache.org/jira/browse/YARN-6207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882210#comment-15882210
]
Bibin A Chundatt commented on YARN-6207:
{quote}
Incase if the comment is regarding Fair Scheduler currently we will handle only
Capacity scheduler cases in this jira
{quote}
I had already mentioned will handle only CS .We have marked component also as
Capacity scheduler.
Fair Scheduler probably we handle in another jira
> Move application can fail when attempt add event is delayed
>
>
> Key: YARN-6207
> URL: https://issues.apache.org/jira/browse/YARN-6207
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: YARN-6207.001.patch, YARN-6207.002.patch,
> YARN-6207.003.patch, YARN-6207.004.patch
>
>
> *Steps to reproduce*
> 1.Submit application and delay attempt add to Scheduler
> (Simulate using debug at EventDispatcher for SchedulerEventDispatcher)
> 2. Call move application to destination queue.
> {noformat}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.preValidateMoveApplication(CapacityScheduler.java:2086)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.moveApplicationAcrossQueue(RMAppManager.java:669)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.moveApplicationAcrossQueues(ClientRMService.java:1231)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.moveApplicationAcrossQueues(ApplicationClientProtocolPBServiceImpl.java:388)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:537)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:522)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:867)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:813)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2659)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1483)
> at org.apache.hadoop.ipc.Client.call(Client.java:1429)
> at org.apache.hadoop.ipc.Client.call(Client.java:1339)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:115)
> at com.sun.proxy.$Proxy7.moveApplicationAcrossQueues(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.moveApplicationAcrossQueues(ApplicationClientProtocolPBClientImpl.java:398)
> ... 16 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6229) resource manager web UI display BUG
[ https://issues.apache.org/jira/browse/YARN-6229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] gehaijiang updated YARN-6229: - Description: resourcemanager web UI display bug： Memory Used -3.44TB Containers Running -2607 VCores Used -2607 Lost Nodes 173 These numbers are not correct。 Cluster Metrics Apps Submitted | Apps Pending | Apps Running | Apps Completed | Containers Running | Memory Used | Memory Total | Memory Reserved |VCores Used | VCores Total | VCores Reserved | Active Nodes | Decommissioned Nodes | Lost Nodes | Unhealthy Nodes | Rebooted Nodes 3027432 0 20 3027412 -2607 -3.44TB 9.70TB 0B -2607 7240 0 181 0 was: resourcemanager web UI display bug： Memory Used -3.44TB Containers Running -2607 VCores Used -2607 Lost Nodes 173 These numbers are not correct。 Cluster Metrics Apps Submitted Apps PendingApps RunningApps Completed Containers Running Memory Used Memory TotalMemory Reserved VCores Used VCores TotalVCores Reserved Active NodesDecommissioned NodesLost Nodes Unhealthy Nodes Rebooted Nodes 3027432 0 20 3027412 -2607 -3.44 TB9.70 TB 0 B -2607 72400 181 0 > resource manager web UI display BUG > - > > Key: YARN-6229 > URL: https://issues.apache.org/jira/browse/YARN-6229 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.1 > Environment: hadoop 2.7.1 >Reporter: gehaijiang > > resourcemanager web UI display bug： > Memory Used -3.44TB > Containers Running -2607 > VCores Used -2607 > Lost Nodes 173 > These numbers are not correct。 > Cluster Metrics > Apps Submitted | Apps Pending | Apps Running |Apps Completed | > Containers Running | Memory Used | Memory Total | Memory Reserved |VCores > Used | VCores Total | VCores Reserved | Active Nodes | Decommissioned > Nodes | Lost Nodes | Unhealthy Nodes | Rebooted Nodes > 3027432 0 20 3027412 -2607 -3.44TB 9.70TB 0B -2607 > 72400 181 0 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6153) keepContainer does not work when AM retry window is set
[ https://issues.apache.org/jira/browse/YARN-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kyungwan nam updated YARN-6153: --- Attachment: YARN-6153.005.patch my source tree was not up-to-date. that's why compile is failed. I'm uploading a new patch which is based on up-to-date source. > keepContainer does not work when AM retry window is set > --- > > Key: YARN-6153 > URL: https://issues.apache.org/jira/browse/YARN-6153 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.1 >Reporter: kyungwan nam > Attachments: YARN-6153.001.patch, YARN-6153.002.patch, > YARN-6153.003.patch, YARN-6153.004.patch, YARN-6153.005.patch > > > yarn.resourcemanager.am.max-attempts has been configured to 2 in my cluster. > I submitted a YARN application (slider app) that keepContainers=true, > attemptFailuresValidityInterval=30. > it did work properly when AM was failed firstly. > all containers launched by previous AM were resynced with new AM (attempt2) > without killing containers. > after 10 minutes, I thought AM failure count was reset by > attemptFailuresValidityInterval (5 minutes). > but, all containers were killed when AM was failed secondly. (new AM attempt3 > was launched properly) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
