[jira] [Created] (YARN-2955) mortbay.log (Slf4jLog.java:info(67)) - Stopped SelectChannelConnector
Rocju created YARN-2955: --- Summary: mortbay.log (Slf4jLog.java:info(67)) - Stopped SelectChannelConnector Key: YARN-2955 URL: https://issues.apache.org/jira/browse/YARN-2955 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.3.0 Environment: cdh5.1.0 Reporter: Rocju 2014-12-12 02:26:55,047 INFO mortbay.log (Slf4jLog.java:info(67)) - Stopped SelectChannelConnector@dcnn2:23188 2014-12-12 02:26:55,052 WARN mortbay.log (Slf4jLog.java:warn(89)) - EXCEPTION java.lang.InterruptedException at java.lang.Object.wait(Native Method) at org.mortbay.io.nio.SelectChannelEndPoint.blockWritable(SelectChannelEndPoint.java:279) at org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:545) at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:639) at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580) at java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java:154) at org.mortbay.jetty.AbstractGenerator$OutputWriter.write(AbstractGenerator.java:904) at org.mortbay.jetty.AbstractGenerator$OutputWriter.write(AbstractGenerator.java:755) at java.io.Writer.write(Writer.java:157) at java.io.PrintWriter.newLine(PrintWriter.java:480) at java.io.PrintWriter.println(PrintWriter.java:629) at org.apache.hadoop.yarn.webapp.hamlet.HamletImpl$EImp._p(HamletImpl.java:110) at org.apache.hadoop.yarn.webapp.hamlet.Hamlet$SCRIPT._(Hamlet.java:454) at org.apache.hadoop.yarn.server.resourcemanager.webapp.AppsBlock.render(AppsBlock.java:119) at org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:66) at org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:76) at org.apache.hadoop.yarn.webapp.View.render(View.java:235) at org.apache.hadoop.yarn.webapp.view.HtmlBlock$Block.subView(HtmlBlock.java:40) at org.apache.hadoop.yarn.webapp.hamlet.Hamlet._(Hamlet.java:30347) at org.apache.hadoop.yarn.server.resourcemanager.webapp.AppsBlockWithMetrics.render(AppsBlockWithMetrics.java:29) at org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:66) at org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:76) at org.apache.hadoop.yarn.webapp.View.render(View.java:235) at org.apache.hadoop.yarn.webapp.view.HtmlPage$Page.subView(HtmlPage.java:49) at org.apache.hadoop.yarn.webapp.hamlet.HamletImpl$EImp._v(HamletImpl.java:117) at org.apache.hadoop.yarn.webapp.hamlet.Hamlet$TD._(Hamlet.java:845) at org.apache.hadoop.yarn.webapp.view.TwoColumnLayout.render(TwoColumnLayout.java:56) at org.apache.hadoop.yarn.webapp.view.HtmlPage.render(HtmlPage.java:82) at org.apache.hadoop.yarn.webapp.Dispatcher.render(Dispatcher.java:197) at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:156) at org.apache.hadoop.yarn.server.resourcemanager.webapp.RMDispatcher.service(RMDispatcher.java:77) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263) at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178) at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1183) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
[jira] [Updated] (YARN-2738) Add FairReservationSystem for FairScheduler
[ https://issues.apache.org/jira/browse/YARN-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anubhav Dhoot updated YARN-2738: Attachment: YARN-2738.004.patch Addressed all feedback Allowed configuring the class names at the global level and only ability to turn on/off reservability at queue level. Disallows reservable and dynamic queues at the same time Additional unit tests verifying the same Add FairReservationSystem for FairScheduler --- Key: YARN-2738 URL: https://issues.apache.org/jira/browse/YARN-2738 Project: Hadoop YARN Issue Type: Sub-task Components: fairscheduler Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Attachments: YARN-2738.001.patch, YARN-2738.002.patch, YARN-2738.003.patch, YARN-2738.004.patch Need to create a FairReservationSystem that will implement ReservationSystem for FairScheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2738) Add FairReservationSystem for FairScheduler
[ https://issues.apache.org/jira/browse/YARN-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14243871#comment-14243871 ] Anubhav Dhoot commented on YARN-2738: - resolveReservationQueueName and handleMoveToPlanQueue will be addressed in the YARN-2881 Add FairReservationSystem for FairScheduler --- Key: YARN-2738 URL: https://issues.apache.org/jira/browse/YARN-2738 Project: Hadoop YARN Issue Type: Sub-task Components: fairscheduler Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Attachments: YARN-2738.001.patch, YARN-2738.002.patch, YARN-2738.003.patch, YARN-2738.004.patch Need to create a FairReservationSystem that will implement ReservationSystem for FairScheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2738) Add FairReservationSystem for FairScheduler
[
https://issues.apache.org/jira/browse/YARN-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14243931#comment-14243931
]
Hadoop QA commented on YARN-2738:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12686809/YARN-2738.004.patch
against trunk revision bda748a.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 15 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6102//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6102//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6102//console
This message is automatically generated.
Add FairReservationSystem for FairScheduler
---
Key: YARN-2738
URL: https://issues.apache.org/jira/browse/YARN-2738
Project: Hadoop YARN
Issue Type: Sub-task
Components: fairscheduler
Reporter: Anubhav Dhoot
Assignee: Anubhav Dhoot
Attachments: YARN-2738.001.patch, YARN-2738.002.patch,
YARN-2738.003.patch, YARN-2738.004.patch
Need to create a FairReservationSystem that will implement ReservationSystem
for FairScheduler
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2356) yarn status command for non-existent application/application attempt/container is too verbose
[ https://issues.apache.org/jira/browse/YARN-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-2356: -- Attachment: 0004-YARN-2356.patch Hi [~devaraj.k] Thank you for the update. I have fixed the warnings yarn status command for non-existent application/application attempt/container is too verbose -- Key: YARN-2356 URL: https://issues.apache.org/jira/browse/YARN-2356 Project: Hadoop YARN Issue Type: Bug Components: client Reporter: Sunil G Assignee: Sunil G Priority: Minor Attachments: 0001-YARN-2356.patch, 0002-YARN-2356.patch, 0003-YARN-2356.patch, 0004-YARN-2356.patch, Yarn-2356.1.patch *yarn application -status* or *applicationattempt -status* or *container status* commands can suppress exception such as ApplicationNotFound, ApplicationAttemptNotFound and ContainerNotFound for non-existent entries in RM or History Server. For example, below exception can be suppressed better sunildev@host-a:~/hadoop/hadoop/bin ./yarn application -status application_1402668848165_0015 No GC_PROFILE is given. Defaults to medium. 14/07/25 16:21:45 INFO client.RMProxy: Connecting to ResourceManager at /10.18.40.77:45022 Exception in thread main org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application with id 'application_1402668848165_0015' doesn't exist in RM. at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:285) at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:145) at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:321) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:607) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:932) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2099) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2095) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1626) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2093) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:101) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:166) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103) at $Proxy12.getApplicationReport(Unknown Source) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:291) at org.apache.hadoop.yarn.client.cli.ApplicationCLI.printApplicationReport(ApplicationCLI.java:428) at org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:153) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.hadoop.yarn.client.cli.ApplicationCLI.main(ApplicationCLI.java:76) Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException): Application with id 'application_1402668848165_0015' doesn't exist in RM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2917) Potential deadlock in AsyncDispatcher when system.exit called in AsyncDispatcher#dispatch and AsyscDispatcher#serviceStop from shutdown hook
[
https://issues.apache.org/jira/browse/YARN-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14243968#comment-14243968
]
Hudson commented on YARN-2917:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #38 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/38/])
YARN-2917. Fixed potential deadlock when system.exit is called in
AsyncDispatcher. Contributed by Rohith Sharmaks (jianhe: rev
614b6afea450ebb897fbb2519c6f02e13b9bd12d)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
* hadoop-yarn-project/CHANGES.txt
Potential deadlock in AsyncDispatcher when system.exit called in
AsyncDispatcher#dispatch and AsyscDispatcher#serviceStop from shutdown hook
Key: YARN-2917
URL: https://issues.apache.org/jira/browse/YARN-2917
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Reporter: Rohith
Assignee: Rohith
Priority: Critical
Fix For: 2.7.0
Attachments: 0001-YARN-2917.patch, 0002-YARN-2917.patch
I encoutered scenario where RM hanged while shutting down and keep on logging
{{2014-12-03 19:32:44,283 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Waiting for AsyncDispatcher to drain.}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2243) Order of arguments for Preconditions.checkNotNull() is wrong in SchedulerApplicationAttempt ctor
[
https://issues.apache.org/jira/browse/YARN-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14243964#comment-14243964
]
Hudson commented on YARN-2243:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #38 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/38/])
YARN-2243. Order of arguments for Preconditions.checkNotNull() is wrong in
(devaraj: rev bda748ac3abf30f6cd4c0e22c80c73396abc59fb)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
Order of arguments for Preconditions.checkNotNull() is wrong in
SchedulerApplicationAttempt ctor
Key: YARN-2243
URL: https://issues.apache.org/jira/browse/YARN-2243
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.5.1
Reporter: Ted Yu
Assignee: Devaraj K
Priority: Minor
Attachments: YARN-2243.patch, YARN-2243.patch
{code}
public SchedulerApplicationAttempt(ApplicationAttemptId
applicationAttemptId,
String user, Queue queue, ActiveUsersManager activeUsersManager,
RMContext rmContext) {
Preconditions.checkNotNull(RMContext should not be null, rmContext);
{code}
Order of arguments is wrong for Preconditions.checkNotNull().
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2917) Potential deadlock in AsyncDispatcher when system.exit called in AsyncDispatcher#dispatch and AsyscDispatcher#serviceStop from shutdown hook
[
https://issues.apache.org/jira/browse/YARN-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244024#comment-14244024
]
Hudson commented on YARN-2917:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #773 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/773/])
YARN-2917. Fixed potential deadlock when system.exit is called in
AsyncDispatcher. Contributed by Rohith Sharmaks (jianhe: rev
614b6afea450ebb897fbb2519c6f02e13b9bd12d)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
Potential deadlock in AsyncDispatcher when system.exit called in
AsyncDispatcher#dispatch and AsyscDispatcher#serviceStop from shutdown hook
Key: YARN-2917
URL: https://issues.apache.org/jira/browse/YARN-2917
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Reporter: Rohith
Assignee: Rohith
Priority: Critical
Fix For: 2.7.0
Attachments: 0001-YARN-2917.patch, 0002-YARN-2917.patch
I encoutered scenario where RM hanged while shutting down and keep on logging
{{2014-12-03 19:32:44,283 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Waiting for AsyncDispatcher to drain.}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2243) Order of arguments for Preconditions.checkNotNull() is wrong in SchedulerApplicationAttempt ctor
[
https://issues.apache.org/jira/browse/YARN-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244020#comment-14244020
]
Hudson commented on YARN-2243:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #773 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/773/])
YARN-2243. Order of arguments for Preconditions.checkNotNull() is wrong in
(devaraj: rev bda748ac3abf30f6cd4c0e22c80c73396abc59fb)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
* hadoop-yarn-project/CHANGES.txt
Order of arguments for Preconditions.checkNotNull() is wrong in
SchedulerApplicationAttempt ctor
Key: YARN-2243
URL: https://issues.apache.org/jira/browse/YARN-2243
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.5.1
Reporter: Ted Yu
Assignee: Devaraj K
Priority: Minor
Attachments: YARN-2243.patch, YARN-2243.patch
{code}
public SchedulerApplicationAttempt(ApplicationAttemptId
applicationAttemptId,
String user, Queue queue, ActiveUsersManager activeUsersManager,
RMContext rmContext) {
Preconditions.checkNotNull(RMContext should not be null, rmContext);
{code}
Order of arguments is wrong for Preconditions.checkNotNull().
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2946) Deadlock in ZKRMStateStore
[
https://issues.apache.org/jira/browse/YARN-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244050#comment-14244050
]
Rohith commented on YARN-2946:
--
Another deadlock detected in different flow after fixing previous deadlock :-(
Basically, by convention all the locks should acquire from
*StateMachine.doTransition() - zkRMStateStore.class* or directly
zkRMStateStore.class , but in {{RMStateStore#isFencedState()}} method locking
order is reversed i.e *zkRMStateStore.class - StateMachine.doTransition()*
{noformat}
Found one Java-level deadlock:
=
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$VerifyActiveStatusThread:
waiting to lock monitor 0x00e55698 (object 0xc0272cb0, a
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine),
which is held by AsyncDispatcher event handler
AsyncDispatcher event handler:
waiting to lock monitor 0x013adcf8 (object 0xc0272b10, a
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore),
which is held by
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$VerifyActiveStatusThread
Java stack information for the threads listed above:
===
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$VerifyActiveStatusThread:
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.getCurrentState(StateMachineFactory.java:442)
- waiting to lock 0xc0272cb0 (a
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.isFencedState(RMStateStore.java:693)
- locked 0xc0272b10 (a
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$VerifyActiveStatusThread.run(ZKRMStateStore.java:1020)
AsyncDispatcher event handler:
at java.lang.Object.wait(Native Method)
- waiting on 0xc0272b10 (a
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithCheck(ZKRMStateStore.java:1043)
- locked 0xc0272b10 (a
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithRetries(ZKRMStateStore.java:1070)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.doMultiWithRetries(ZKRMStateStore.java:906)
- locked 0xc0272b10 (a
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.doMultiWithRetries(ZKRMStateStore.java:920)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.createWithRetries(ZKRMStateStore.java:929)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.storeApplicationStateInternal(ZKRMStateStore.java:608)
- locked 0xc0272b10 (a
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$StoreAppTransition.transition(RMStateStore.java:146)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$StoreAppTransition.transition(RMStateStore.java:131)
at
org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362)
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
- locked 0xc0272cb0 (a
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.handleStoreEvent(RMStateStore.java:699)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ForwardingEventHandler.handle(RMStateStore.java:754)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ForwardingEventHandler.handle(RMStateStore.java:749)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
at java.lang.Thread.run(Thread.java:745)
Found 1 deadlock.
{noformat}
Deadlock in ZKRMStateStore
--
Key: YARN-2946
URL:
[jira] [Created] (YARN-2956) Some yarn-site index linked pages are difficult to discover because are not in the side bar
Remus Rusanu created YARN-2956: -- Summary: Some yarn-site index linked pages are difficult to discover because are not in the side bar Key: YARN-2956 URL: https://issues.apache.org/jira/browse/YARN-2956 Project: Hadoop YARN Issue Type: Bug Components: documentation Affects Versions: 2.6.0 Reporter: Remus Rusanu Priority: Minor The yarn-site index.apt.vm page is difficult to 'stumble upon' because the hadoop.apache.org/ sidebar navigation does not link to it. One needs to know the URL http://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-site/ to land on it. The links from the index page do not match the links from the side bar, so some pages are quickly accessible (from sidebar). I propose the links from the index.apt.vm to match the links from the YARN side bar subsection (Ideally through one single definition file, but I don't understand the APT generation process well enough to call out how this can be achieved. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2946) Deadlock in ZKRMStateStore
[
https://issues.apache.org/jira/browse/YARN-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244098#comment-14244098
]
Rohith commented on YARN-2946:
--
Above 2 deadlock's can be directly fixed by removing *synchronized* keyword
which was not really required.
But I see there many other potential deadlocks which can appear easily in the
following methods. Below all methods does reverse locking i.e
*zkRMStateStore.class - StateMachine.doTransition()* through
{{RMStateStore#isFencedState()}}
# Method {{RMStateStore#storeRMDTMasterKey()}}
# Method {{RMStateStore#removeRMDTMasterKey()}}
# Method {{RMStateStore#storeRMDelegationTokenAndSequenceNumber()}}
# Method {{RMStateStore#removeRMDelegationToken()}}
# Method {{RMStateStore#updateRMDelegationTokenAndSequenceNumber()}}
# Method {{ZKRMStateStore#storeOrUpdateAMRMTokenSecretManagerState()}}
So only fixing 1 or 2 deadlock flows does not really fix other potential dead
lock issues.
*I propose following solution to handle all these deadlock flows*
Option-1 :
# For all above mentioned method's causing deadlock , introduce StateMachine in
RMStateStore like handling application store. So all the execution flows from
StateMachine-zkRMStateStore.class.
# Along with 1st , StateMachine should be guarded with Read-Write lock.
Option-2 :
# Fix the visible eadlocks i.e 2 found in this jira. And Option-1 do in
separate improvement task.
Handling all the deadlock flows, i would like to do in one umbrella jira. This
is to ensure we do not miss any these deadlock flows.
Please let me your suggestions/thoughts?
Deadlock in ZKRMStateStore
--
Key: YARN-2946
URL: https://issues.apache.org/jira/browse/YARN-2946
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.7.0
Reporter: Rohith
Assignee: Rohith
Priority: Blocker
Attachments: 0001-YARN-2946.patch, 0002-YARN-2946.patch,
TestYARN2946.java
Found one deadlock in ZKRMStateStore.
# Initial stage zkClient is null because of zk disconnected event.
# When ZKRMstatestore#runWithCheck() wait(zkSessionTimeout) for zkClient to
re establish zookeeper connection either via synconnected or expired event,
it is highly possible that any other thred can obtain lock on
{{ZKRMStateStore.this}} from state machine transition events. This cause
Deadlock in ZKRMStateStore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2946) DeadLock's in RMStateStore-ZKRMStateStore
[
https://issues.apache.org/jira/browse/YARN-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith updated YARN-2946:
-
Summary: DeadLock's in RMStateStore-ZKRMStateStore (was: Deadlock in
ZKRMStateStore)
DeadLock's in RMStateStore-ZKRMStateStore
---
Key: YARN-2946
URL: https://issues.apache.org/jira/browse/YARN-2946
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.7.0
Reporter: Rohith
Assignee: Rohith
Priority: Blocker
Attachments: 0001-YARN-2946.patch, 0002-YARN-2946.patch,
TestYARN2946.java
Found one deadlock in ZKRMStateStore.
# Initial stage zkClient is null because of zk disconnected event.
# When ZKRMstatestore#runWithCheck() wait(zkSessionTimeout) for zkClient to
re establish zookeeper connection either via synconnected or expired event,
it is highly possible that any other thred can obtain lock on
{{ZKRMStateStore.this}} from state machine transition events. This cause
Deadlock in ZKRMStateStore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2946) DeadLock's in RMStateStore-ZKRMStateStore
[
https://issues.apache.org/jira/browse/YARN-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244142#comment-14244142
]
Varun Saxena commented on YARN-2946:
[~rohithsharma], good catch. As you said {{synchronized}} can be removed from
{{RMStateStore#isFencedState()}}.
Regarding the methods you listed out, I think we can avoid reverse locking i.e.
*ZKRMStateStore.class - StateMachine.doTransition()* by making the following
changes.
# Remove {{synchronized}} keyword from each one of the methods listed above.
# Separate out State machine synchronization(invoked by call to
{{isFencedState}} {{notifyStoreOperationFailed}}) and ZKRMStateStore
synchronization by putting the relevant code in a synchronized block.
For example, code for RMStateStore#storeRMDTMasterKey() can be changed as under
:
{code:title=RMStateStore.java|borderStyle=solid}
public void storeRMDTMasterKey(DelegationKey delegationKey) {
if(isFencedState()) {
LOG.info(State store is in Fenced state. Can't store RM Delegation +
Token Master key.);
return;
}
try {
synchronized(this) {
storeRMDTMasterKeyState(delegationKey);
}
} catch (Exception e) {
notifyStoreOperationFailed(e);
}
}
{code}
DeadLock's in RMStateStore-ZKRMStateStore
---
Key: YARN-2946
URL: https://issues.apache.org/jira/browse/YARN-2946
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.7.0
Reporter: Rohith
Assignee: Rohith
Priority: Blocker
Attachments: 0001-YARN-2946.patch, 0002-YARN-2946.patch,
TestYARN2946.java
Found one deadlock in ZKRMStateStore.
# Initial stage zkClient is null because of zk disconnected event.
# When ZKRMstatestore#runWithCheck() wait(zkSessionTimeout) for zkClient to
re establish zookeeper connection either via synconnected or expired event,
it is highly possible that any other thred can obtain lock on
{{ZKRMStateStore.this}} from state machine transition events. This cause
Deadlock in ZKRMStateStore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1842) InvalidApplicationMasterRequestException raised during AM-requested shutdown
[ https://issues.apache.org/jira/browse/YARN-1842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244176#comment-14244176 ] Falk Scheerschmidt commented on YARN-1842: -- Same problem here with Hadoop 2.6 and Zookeeper 3.4 on Debian. InvalidApplicationMasterRequestException raised during AM-requested shutdown Key: YARN-1842 URL: https://issues.apache.org/jira/browse/YARN-1842 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.3.0 Reporter: Steve Loughran Priority: Minor Attachments: hoyalogs.tar.gz Report of the RM raising a stack trace [https://gist.github.com/matyix/9596735] during AM-initiated shutdown. The AM could just swallow this and exit, but it could be a sign of a race condition YARN-side, or maybe just in the RM client code/AM dual signalling the shutdown. I haven't replicated this myself; maybe the stack will help track down the problem. Otherwise: what is the policy YARN apps should adopt for AM's handling errors on shutdown? go straight to an exit(-1)? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2243) Order of arguments for Preconditions.checkNotNull() is wrong in SchedulerApplicationAttempt ctor
[
https://issues.apache.org/jira/browse/YARN-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244190#comment-14244190
]
Hudson commented on YARN-2243:
--
SUCCESS: Integrated in Hadoop-Hdfs-trunk #1970 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1970/])
YARN-2243. Order of arguments for Preconditions.checkNotNull() is wrong in
(devaraj: rev bda748ac3abf30f6cd4c0e22c80c73396abc59fb)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
* hadoop-yarn-project/CHANGES.txt
Order of arguments for Preconditions.checkNotNull() is wrong in
SchedulerApplicationAttempt ctor
Key: YARN-2243
URL: https://issues.apache.org/jira/browse/YARN-2243
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.5.1
Reporter: Ted Yu
Assignee: Devaraj K
Priority: Minor
Attachments: YARN-2243.patch, YARN-2243.patch
{code}
public SchedulerApplicationAttempt(ApplicationAttemptId
applicationAttemptId,
String user, Queue queue, ActiveUsersManager activeUsersManager,
RMContext rmContext) {
Preconditions.checkNotNull(RMContext should not be null, rmContext);
{code}
Order of arguments is wrong for Preconditions.checkNotNull().
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2243) Order of arguments for Preconditions.checkNotNull() is wrong in SchedulerApplicationAttempt ctor
[
https://issues.apache.org/jira/browse/YARN-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244199#comment-14244199
]
Hudson commented on YARN-2243:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #36 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/36/])
YARN-2243. Order of arguments for Preconditions.checkNotNull() is wrong in
(devaraj: rev bda748ac3abf30f6cd4c0e22c80c73396abc59fb)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
Order of arguments for Preconditions.checkNotNull() is wrong in
SchedulerApplicationAttempt ctor
Key: YARN-2243
URL: https://issues.apache.org/jira/browse/YARN-2243
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.5.1
Reporter: Ted Yu
Assignee: Devaraj K
Priority: Minor
Attachments: YARN-2243.patch, YARN-2243.patch
{code}
public SchedulerApplicationAttempt(ApplicationAttemptId
applicationAttemptId,
String user, Queue queue, ActiveUsersManager activeUsersManager,
RMContext rmContext) {
Preconditions.checkNotNull(RMContext should not be null, rmContext);
{code}
Order of arguments is wrong for Preconditions.checkNotNull().
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2917) Potential deadlock in AsyncDispatcher when system.exit called in AsyncDispatcher#dispatch and AsyscDispatcher#serviceStop from shutdown hook
[
https://issues.apache.org/jira/browse/YARN-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244203#comment-14244203
]
Hudson commented on YARN-2917:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #36 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/36/])
YARN-2917. Fixed potential deadlock when system.exit is called in
AsyncDispatcher. Contributed by Rohith Sharmaks (jianhe: rev
614b6afea450ebb897fbb2519c6f02e13b9bd12d)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
* hadoop-yarn-project/CHANGES.txt
Potential deadlock in AsyncDispatcher when system.exit called in
AsyncDispatcher#dispatch and AsyscDispatcher#serviceStop from shutdown hook
Key: YARN-2917
URL: https://issues.apache.org/jira/browse/YARN-2917
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Reporter: Rohith
Assignee: Rohith
Priority: Critical
Fix For: 2.7.0
Attachments: 0001-YARN-2917.patch, 0002-YARN-2917.patch
I encoutered scenario where RM hanged while shutting down and keep on logging
{{2014-12-03 19:32:44,283 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Waiting for AsyncDispatcher to drain.}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2917) Potential deadlock in AsyncDispatcher when system.exit called in AsyncDispatcher#dispatch and AsyscDispatcher#serviceStop from shutdown hook
[
https://issues.apache.org/jira/browse/YARN-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244194#comment-14244194
]
Hudson commented on YARN-2917:
--
SUCCESS: Integrated in Hadoop-Hdfs-trunk #1970 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1970/])
YARN-2917. Fixed potential deadlock when system.exit is called in
AsyncDispatcher. Contributed by Rohith Sharmaks (jianhe: rev
614b6afea450ebb897fbb2519c6f02e13b9bd12d)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
Potential deadlock in AsyncDispatcher when system.exit called in
AsyncDispatcher#dispatch and AsyscDispatcher#serviceStop from shutdown hook
Key: YARN-2917
URL: https://issues.apache.org/jira/browse/YARN-2917
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Reporter: Rohith
Assignee: Rohith
Priority: Critical
Fix For: 2.7.0
Attachments: 0001-YARN-2917.patch, 0002-YARN-2917.patch
I encoutered scenario where RM hanged while shutting down and keep on logging
{{2014-12-03 19:32:44,283 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Waiting for AsyncDispatcher to drain.}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2946) DeadLock's in RMStateStore-ZKRMStateStore
[
https://issues.apache.org/jira/browse/YARN-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244225#comment-14244225
]
Varun Saxena commented on YARN-2946:
Frankly, as all of these methods *merely have a single line calling another
method* in {{ZKRMStateStore}}, in addition to call to isFencedState and
notifyStoreOperationFailed. Hence, we *do not even need a synchronized block*
in these methods in RMStateStore. Just make the relevant method in
ZKRMStateStore *synchronized*.
DeadLock's in RMStateStore-ZKRMStateStore
---
Key: YARN-2946
URL: https://issues.apache.org/jira/browse/YARN-2946
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.7.0
Reporter: Rohith
Assignee: Rohith
Priority: Blocker
Attachments: 0001-YARN-2946.patch, 0002-YARN-2946.patch,
TestYARN2946.java
Found one deadlock in ZKRMStateStore.
# Initial stage zkClient is null because of zk disconnected event.
# When ZKRMstatestore#runWithCheck() wait(zkSessionTimeout) for zkClient to
re establish zookeeper connection either via synconnected or expired event,
it is highly possible that any other thred can obtain lock on
{{ZKRMStateStore.this}} from state machine transition events. This cause
Deadlock in ZKRMStateStore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2917) Potential deadlock in AsyncDispatcher when system.exit called in AsyncDispatcher#dispatch and AsyscDispatcher#serviceStop from shutdown hook
[
https://issues.apache.org/jira/browse/YARN-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244246#comment-14244246
]
Hudson commented on YARN-2917:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #40 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/40/])
YARN-2917. Fixed potential deadlock when system.exit is called in
AsyncDispatcher. Contributed by Rohith Sharmaks (jianhe: rev
614b6afea450ebb897fbb2519c6f02e13b9bd12d)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
* hadoop-yarn-project/CHANGES.txt
Potential deadlock in AsyncDispatcher when system.exit called in
AsyncDispatcher#dispatch and AsyscDispatcher#serviceStop from shutdown hook
Key: YARN-2917
URL: https://issues.apache.org/jira/browse/YARN-2917
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Reporter: Rohith
Assignee: Rohith
Priority: Critical
Fix For: 2.7.0
Attachments: 0001-YARN-2917.patch, 0002-YARN-2917.patch
I encoutered scenario where RM hanged while shutting down and keep on logging
{{2014-12-03 19:32:44,283 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Waiting for AsyncDispatcher to drain.}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2243) Order of arguments for Preconditions.checkNotNull() is wrong in SchedulerApplicationAttempt ctor
[
https://issues.apache.org/jira/browse/YARN-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244242#comment-14244242
]
Hudson commented on YARN-2243:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #40 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/40/])
YARN-2243. Order of arguments for Preconditions.checkNotNull() is wrong in
(devaraj: rev bda748ac3abf30f6cd4c0e22c80c73396abc59fb)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
* hadoop-yarn-project/CHANGES.txt
Order of arguments for Preconditions.checkNotNull() is wrong in
SchedulerApplicationAttempt ctor
Key: YARN-2243
URL: https://issues.apache.org/jira/browse/YARN-2243
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.5.1
Reporter: Ted Yu
Assignee: Devaraj K
Priority: Minor
Attachments: YARN-2243.patch, YARN-2243.patch
{code}
public SchedulerApplicationAttempt(ApplicationAttemptId
applicationAttemptId,
String user, Queue queue, ActiveUsersManager activeUsersManager,
RMContext rmContext) {
Preconditions.checkNotNull(RMContext should not be null, rmContext);
{code}
Order of arguments is wrong for Preconditions.checkNotNull().
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2243) Order of arguments for Preconditions.checkNotNull() is wrong in SchedulerApplicationAttempt ctor
[
https://issues.apache.org/jira/browse/YARN-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244269#comment-14244269
]
Hudson commented on YARN-2243:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1990 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1990/])
YARN-2243. Order of arguments for Preconditions.checkNotNull() is wrong in
(devaraj: rev bda748ac3abf30f6cd4c0e22c80c73396abc59fb)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
* hadoop-yarn-project/CHANGES.txt
Order of arguments for Preconditions.checkNotNull() is wrong in
SchedulerApplicationAttempt ctor
Key: YARN-2243
URL: https://issues.apache.org/jira/browse/YARN-2243
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.5.1
Reporter: Ted Yu
Assignee: Devaraj K
Priority: Minor
Attachments: YARN-2243.patch, YARN-2243.patch
{code}
public SchedulerApplicationAttempt(ApplicationAttemptId
applicationAttemptId,
String user, Queue queue, ActiveUsersManager activeUsersManager,
RMContext rmContext) {
Preconditions.checkNotNull(RMContext should not be null, rmContext);
{code}
Order of arguments is wrong for Preconditions.checkNotNull().
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2917) Potential deadlock in AsyncDispatcher when system.exit called in AsyncDispatcher#dispatch and AsyscDispatcher#serviceStop from shutdown hook
[
https://issues.apache.org/jira/browse/YARN-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244273#comment-14244273
]
Hudson commented on YARN-2917:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1990 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1990/])
YARN-2917. Fixed potential deadlock when system.exit is called in
AsyncDispatcher. Contributed by Rohith Sharmaks (jianhe: rev
614b6afea450ebb897fbb2519c6f02e13b9bd12d)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
Potential deadlock in AsyncDispatcher when system.exit called in
AsyncDispatcher#dispatch and AsyscDispatcher#serviceStop from shutdown hook
Key: YARN-2917
URL: https://issues.apache.org/jira/browse/YARN-2917
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Reporter: Rohith
Assignee: Rohith
Priority: Critical
Fix For: 2.7.0
Attachments: 0001-YARN-2917.patch, 0002-YARN-2917.patch
I encoutered scenario where RM hanged while shutting down and keep on logging
{{2014-12-03 19:32:44,283 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Waiting for AsyncDispatcher to drain.}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2952) Incorrect version check in RMStateStore
[
https://issues.apache.org/jira/browse/YARN-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1427#comment-1427
]
Rohith commented on YARN-2952:
--
+1 for issue, always base major version is considered as 1. But in real use
cases, base major version can be greater than 2 or 3 which installation itself
will fail.
Incorrect version check in RMStateStore
---
Key: YARN-2952
URL: https://issues.apache.org/jira/browse/YARN-2952
Project: Hadoop YARN
Issue Type: Bug
Reporter: Jian He
Assignee: Rohith
In RMStateStore#checkVersion: if we modify tCURRENT_VERSION_INFO to 2.0,
it'll still store the version as 1.0 which is incorrect; The same thing might
happen to NM store, timeline store.
{code}
// if there is no version info, treat it as 1.0;
if (loadedVersion == null) {
loadedVersion = Version.newInstance(1, 0);
}
if (loadedVersion.isCompatibleTo(getCurrentVersion())) {
LOG.info(Storing RM state version info + getCurrentVersion());
storeVersion();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2912) Jersey Tests failing with port in use
[ https://issues.apache.org/jira/browse/YARN-2912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244470#comment-14244470 ] Hudson commented on YARN-2912: -- FAILURE: Integrated in Hadoop-trunk-Commit #6706 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6706/]) YARN-2912 Jersey Tests failing with port in use. (varun saxena via stevel) (stevel: rev 3681de203949f84c9fa4a6df49948dbf6980c9ba) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesNodes.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesDelegationTokens.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/webapp/TestTimelineWebServices.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesFairScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesNodeLabels.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServicesContainers.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesAppsModification.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServicesApps.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/TestAHSWebServices.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/webapp/JerseyTestBase.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServices.java * hadoop-yarn-project/CHANGES.txt Jersey Tests failing with port in use - Key: YARN-2912 URL: https://issues.apache.org/jira/browse/YARN-2912 Project: Hadoop YARN Issue Type: Bug Components: test Affects Versions: 3.0.0 Environment: jenkins on java 8 Reporter: Steve Loughran Assignee: Varun Saxena Fix For: 2.7.0 Attachments: YARN-2912.patch Jersey tests like TestNMWebServices apps are failing with port in use. The jersey test runner appears to always use the same port unless a system property is set to point to a different one. Every test should really be changing that sysprop in a @Before method -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2952) Incorrect version check in RMStateStore
[
https://issues.apache.org/jira/browse/YARN-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244491#comment-14244491
]
Zhijie Shen commented on YARN-2952:
---
This problematic code has be propagated to multiple places where leveldb is
used, which need to be fixed together.
Incorrect version check in RMStateStore
---
Key: YARN-2952
URL: https://issues.apache.org/jira/browse/YARN-2952
Project: Hadoop YARN
Issue Type: Bug
Reporter: Jian He
Assignee: Rohith
In RMStateStore#checkVersion: if we modify tCURRENT_VERSION_INFO to 2.0,
it'll still store the version as 1.0 which is incorrect; The same thing might
happen to NM store, timeline store.
{code}
// if there is no version info, treat it as 1.0;
if (loadedVersion == null) {
loadedVersion = Version.newInstance(1, 0);
}
if (loadedVersion.isCompatibleTo(getCurrentVersion())) {
LOG.info(Storing RM state version info + getCurrentVersion());
storeVersion();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2912) Jersey Tests failing with port in use
[ https://issues.apache.org/jira/browse/YARN-2912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244551#comment-14244551 ] Hudson commented on YARN-2912: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #39 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/39/]) YARN-2912 Jersey Tests failing with port in use. (varun saxena via stevel) (stevel: rev 3681de203949f84c9fa4a6df49948dbf6980c9ba) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/webapp/TestTimelineWebServices.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesNodeLabels.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesAppsModification.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/TestAHSWebServices.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesFairScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServices.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/webapp/JerseyTestBase.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesDelegationTokens.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServicesContainers.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServicesApps.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesNodes.java Jersey Tests failing with port in use - Key: YARN-2912 URL: https://issues.apache.org/jira/browse/YARN-2912 Project: Hadoop YARN Issue Type: Bug Components: test Affects Versions: 3.0.0 Environment: jenkins on java 8 Reporter: Steve Loughran Assignee: Varun Saxena Fix For: 2.7.0 Attachments: YARN-2912.patch Jersey tests like TestNMWebServices apps are failing with port in use. The jersey test runner appears to always use the same port unless a system property is set to point to a different one. Every test should really be changing that sysprop in a @Before method -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2938) Fix new findbugs warnings in hadoop-yarn-resourcemanager and hadoop-yarn-applicationhistoryservice
[ https://issues.apache.org/jira/browse/YARN-2938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-2938: --- Attachment: FindBugs Report.html Fix new findbugs warnings in hadoop-yarn-resourcemanager and hadoop-yarn-applicationhistoryservice -- Key: YARN-2938 URL: https://issues.apache.org/jira/browse/YARN-2938 Project: Hadoop YARN Issue Type: Improvement Reporter: Varun Saxena Assignee: Varun Saxena Fix For: 2.7.0 Attachments: FindBugs Report.html, YARN-2938.001.patch, YARN-2938.002.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2938) Fix new findbugs warnings in hadoop-yarn-resourcemanager and hadoop-yarn-applicationhistoryservice
[ https://issues.apache.org/jira/browse/YARN-2938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244568#comment-14244568 ] Varun Saxena commented on YARN-2938: Report for RM attached. Inconsistent Synchronization I will verify once again. Fix new findbugs warnings in hadoop-yarn-resourcemanager and hadoop-yarn-applicationhistoryservice -- Key: YARN-2938 URL: https://issues.apache.org/jira/browse/YARN-2938 Project: Hadoop YARN Issue Type: Improvement Reporter: Varun Saxena Assignee: Varun Saxena Fix For: 2.7.0 Attachments: FindBugs Report.html, YARN-2938.001.patch, YARN-2938.002.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2938) Fix new findbugs warnings in hadoop-yarn-resourcemanager and hadoop-yarn-applicationhistoryservice
[
https://issues.apache.org/jira/browse/YARN-2938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244567#comment-14244567
]
Hadoop QA commented on YARN-2938:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12686669/YARN-2938.002.patch
against trunk revision 3681de2.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6103//console
This message is automatically generated.
Fix new findbugs warnings in hadoop-yarn-resourcemanager and
hadoop-yarn-applicationhistoryservice
--
Key: YARN-2938
URL: https://issues.apache.org/jira/browse/YARN-2938
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Varun Saxena
Assignee: Varun Saxena
Fix For: 2.7.0
Attachments: FindBugs Report.html, YARN-2938.001.patch,
YARN-2938.002.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2850) DistributedShell: Add support for disk I/O request
[ https://issues.apache.org/jira/browse/YARN-2850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-2850: -- Attachment: YARN-2850-1.patch Attach the first patch for review, based on YARN-2618. DistributedShell: Add support for disk I/O request -- Key: YARN-2850 URL: https://issues.apache.org/jira/browse/YARN-2850 Project: Hadoop YARN Issue Type: Sub-task Reporter: Wei Yan Assignee: Wei Yan Attachments: YARN-2850-1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2851) YarnClient: Add support for disk I/O resource/request information
[ https://issues.apache.org/jira/browse/YARN-2851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-2851: -- Attachment: YARN-2851-1.patch Put an initial patch for review, based on YARN-2618. YarnClient: Add support for disk I/O resource/request information - Key: YARN-2851 URL: https://issues.apache.org/jira/browse/YARN-2851 Project: Hadoop YARN Issue Type: Sub-task Reporter: Wei Yan Assignee: Wei Yan Attachments: YARN-2851-1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2852) WebUI Metrics: Add disk I/O resource information to the web ui and metrics
[ https://issues.apache.org/jira/browse/YARN-2852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-2852: -- Summary: WebUI Metrics: Add disk I/O resource information to the web ui and metrics (was: WebUI: Add disk I/O resource information to the web ui) WebUI Metrics: Add disk I/O resource information to the web ui and metrics Key: YARN-2852 URL: https://issues.apache.org/jira/browse/YARN-2852 Project: Hadoop YARN Issue Type: Sub-task Reporter: Wei Yan Assignee: Wei Yan Attachments: YARN-2852-1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2852) WebUI Metrics: Add disk I/O resource information to the web ui and metrics
[ https://issues.apache.org/jira/browse/YARN-2852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-2852: -- Attachment: YARN-2852-1.patch Patch for review, based on YARN-2618. WebUI Metrics: Add disk I/O resource information to the web ui and metrics Key: YARN-2852 URL: https://issues.apache.org/jira/browse/YARN-2852 Project: Hadoop YARN Issue Type: Sub-task Reporter: Wei Yan Assignee: Wei Yan Attachments: YARN-2852-1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2957) Create unit test to automatically compare YarnConfiguration and yarn-default.xml
Ray Chiang created YARN-2957: Summary: Create unit test to automatically compare YarnConfiguration and yarn-default.xml Key: YARN-2957 URL: https://issues.apache.org/jira/browse/YARN-2957 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.6.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Create a unit test that will automatically compare the fields in YarnConfiguration and yarn-default.xml. It should throw an error if a property is missing in either the class or the file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2937) Fix new findbugs warnings in hadoop-yarn-nodemanager
[
https://issues.apache.org/jira/browse/YARN-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244628#comment-14244628
]
Zhijie Shen commented on YARN-2937:
---
1. What's the problem around Random?
{code}
-Random r = new Random();
-long randomPosition = Math.abs(r.nextLong()) % totalAvailable;
+long randomPosition = Math.abs(RandomUtils.nextLong()) % totalAvailable;
{code}
2. Can we use IOUtils?
{code}
- // Close the streams
- try {
-in.close();
- } catch (IOException e2) {
-LOG.warn(Error closing the stream: + getMtabFileName(), e2);
+ if(in != null) {
+// Close the streams
+try {
+ in.close();
+} catch (IOException e2) {
+ LOG.warn(Error closing the stream: + getMtabFileName(), e2);
+}
{code}
3. Is this removed because bufReader will consequently close fileReader too?
{code}
- if (fileReader != null) {
-fileReader.close();
- }
{code}
4. Again, would you mind explaining a bit about what have been excluded in
findbugs-exclude.xml? Thanks!
Fix new findbugs warnings in hadoop-yarn-nodemanager
Key: YARN-2937
URL: https://issues.apache.org/jira/browse/YARN-2937
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Varun Saxena
Assignee: Varun Saxena
Fix For: 2.7.0
Attachments: HADOOP-11373.patch, YARN-2937.001.patch,
YARN-2937.002.patch, YARN-2937.003.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2950) Change message to mandate, not suggest JS requirement on UI
[
https://issues.apache.org/jira/browse/YARN-2950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dustin Cote updated YARN-2950:
--
Attachment: YARN-2950-1.patch
Updating the error message as Harsh suggested.
Change message to mandate, not suggest JS requirement on UI
---
Key: YARN-2950
URL: https://issues.apache.org/jira/browse/YARN-2950
Project: Hadoop YARN
Issue Type: Improvement
Components: webapp
Affects Versions: 2.5.0
Reporter: Harsh J
Assignee: Dustin Cote
Priority: Minor
Labels: newbie
Attachments: YARN-2950-1.patch
Most of YARN's UIs do not work with JavaScript disabled on the browser, cause
they appear to send back data as JS arrays instead of within the actual HTML
content.
The JQueryUI prints only a mild warning about this suggesting that {{This
page works best with javascript enabled.}}, when in fact it ought to be
{{This page will not function without javascript enabled. Please enable
javascript on your browser.}} or something as such (more direct).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2620) FairScheduler: Add disk I/O resource to the DRF implementation
[ https://issues.apache.org/jira/browse/YARN-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-2620: -- Attachment: YARN-2620-1.patch FairScheduler: Add disk I/O resource to the DRF implementation -- Key: YARN-2620 URL: https://issues.apache.org/jira/browse/YARN-2620 Project: Hadoop YARN Issue Type: Sub-task Reporter: Wei Yan Assignee: Wei Yan Attachments: YARN-2620-1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2620) FairScheduler: Add disk I/O resource to the DRF implementation
[ https://issues.apache.org/jira/browse/YARN-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244664#comment-14244664 ] Wei Yan commented on YARN-2620: --- Initial patch, based on YARN-2168. Some testcases fail, should be ok after we revisit the configuration in YARN-2941. FairScheduler: Add disk I/O resource to the DRF implementation -- Key: YARN-2620 URL: https://issues.apache.org/jira/browse/YARN-2620 Project: Hadoop YARN Issue Type: Sub-task Reporter: Wei Yan Assignee: Wei Yan Attachments: YARN-2620-1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2620) FairScheduler: Add disk I/O resource to the DRF implementation
[ https://issues.apache.org/jira/browse/YARN-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244666#comment-14244666 ] Wei Yan commented on YARN-2620: --- typo... should be YARN-2618. FairScheduler: Add disk I/O resource to the DRF implementation -- Key: YARN-2620 URL: https://issues.apache.org/jira/browse/YARN-2620 Project: Hadoop YARN Issue Type: Sub-task Reporter: Wei Yan Assignee: Wei Yan Attachments: YARN-2620-1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2957) Create unit test to automatically compare YarnConfiguration and yarn-default.xml
[ https://issues.apache.org/jira/browse/YARN-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated YARN-2957: - Attachment: HADOOP-11399.002.patch Cleaned up the javadoc. Create unit test to automatically compare YarnConfiguration and yarn-default.xml Key: YARN-2957 URL: https://issues.apache.org/jira/browse/YARN-2957 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.6.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: supportability Create a unit test that will automatically compare the fields in YarnConfiguration and yarn-default.xml. It should throw an error if a property is missing in either the class or the file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2957) Create unit test to automatically compare YarnConfiguration and yarn-default.xml
[ https://issues.apache.org/jira/browse/YARN-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated YARN-2957: - Attachment: (was: HADOOP-11399.002.patch) Create unit test to automatically compare YarnConfiguration and yarn-default.xml Key: YARN-2957 URL: https://issues.apache.org/jira/browse/YARN-2957 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.6.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: supportability Create a unit test that will automatically compare the fields in YarnConfiguration and yarn-default.xml. It should throw an error if a property is missing in either the class or the file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2957) Create unit test to automatically compare YarnConfiguration and yarn-default.xml
[ https://issues.apache.org/jira/browse/YARN-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated YARN-2957: - Attachment: YARN-2957.001.patch Create unit test to automatically compare YarnConfiguration and yarn-default.xml Key: YARN-2957 URL: https://issues.apache.org/jira/browse/YARN-2957 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.6.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: supportability Attachments: YARN-2957.001.patch Create a unit test that will automatically compare the fields in YarnConfiguration and yarn-default.xml. It should throw an error if a property is missing in either the class or the file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2950) Change message to mandate, not suggest JS requirement on UI
[
https://issues.apache.org/jira/browse/YARN-2950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244712#comment-14244712
]
Hadoop QA commented on YARN-2950:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12686924/YARN-2950-1.patch
against trunk revision 3681de2.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 25 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6104//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6104//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6104//console
This message is automatically generated.
Change message to mandate, not suggest JS requirement on UI
---
Key: YARN-2950
URL: https://issues.apache.org/jira/browse/YARN-2950
Project: Hadoop YARN
Issue Type: Improvement
Components: webapp
Affects Versions: 2.5.0
Reporter: Harsh J
Assignee: Dustin Cote
Priority: Minor
Labels: newbie
Attachments: YARN-2950-1.patch
Most of YARN's UIs do not work with JavaScript disabled on the browser, cause
they appear to send back data as JS arrays instead of within the actual HTML
content.
The JQueryUI prints only a mild warning about this suggesting that {{This
page works best with javascript enabled.}}, when in fact it ought to be
{{This page will not function without javascript enabled. Please enable
javascript on your browser.}} or something as such (more direct).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2944) SCMStore/InMemorySCMStore is not currently compatible with ReflectionUtils#newInstance
[
https://issues.apache.org/jira/browse/YARN-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Trezzo updated YARN-2944:
---
Attachment: YARN-2944-trunk-v2.patch
Attached is V2 to address [~sjlee0]'s comments and to fix unit tests.
SCMStore/InMemorySCMStore is not currently compatible with
ReflectionUtils#newInstance
--
Key: YARN-2944
URL: https://issues.apache.org/jira/browse/YARN-2944
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Chris Trezzo
Assignee: Chris Trezzo
Priority: Minor
Attachments: YARN-2944-trunk-v1.patch, YARN-2944-trunk-v2.patch
Currently the Shared Cache Manager uses ReflectionUtils#newInstance to create
the SCMStore service. Unfortunately the SCMStore class does not have a
0-argument constructor.
On startup, the SCM fails with the following:
{noformat}
14/12/09 16:10:53 INFO service.AbstractService: Service SharedCacheManager
failed in state INITED; cause: java.lang.RuntimeException:
java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
java.lang.RuntimeException: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156)
Caused by: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
at java.lang.Class.getConstructor0(Class.java:2763)
at java.lang.Class.getDeclaredConstructor(Class.java:2021)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
... 4 more
14/12/09 16:10:53 FATAL sharedcachemanager.SharedCacheManager: Error starting
SharedCacheManager
java.lang.RuntimeException: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156)
Caused by: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
at java.lang.Class.getConstructor0(Class.java:2763)
at java.lang.Class.getDeclaredConstructor(Class.java:2021)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
... 4 more
{noformat}
This JIRA is to add a 0-argument constructor to SCMStore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2944) SCMStore/InMemorySCMStore is not currently compatible with ReflectionUtils#newInstance
[
https://issues.apache.org/jira/browse/YARN-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Trezzo updated YARN-2944:
---
Attachment: YARN-2944-trunk-v2.patch
SCMStore/InMemorySCMStore is not currently compatible with
ReflectionUtils#newInstance
--
Key: YARN-2944
URL: https://issues.apache.org/jira/browse/YARN-2944
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Chris Trezzo
Assignee: Chris Trezzo
Priority: Minor
Attachments: YARN-2944-trunk-v1.patch, YARN-2944-trunk-v2.patch
Currently the Shared Cache Manager uses ReflectionUtils#newInstance to create
the SCMStore service. Unfortunately the SCMStore class does not have a
0-argument constructor.
On startup, the SCM fails with the following:
{noformat}
14/12/09 16:10:53 INFO service.AbstractService: Service SharedCacheManager
failed in state INITED; cause: java.lang.RuntimeException:
java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
java.lang.RuntimeException: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156)
Caused by: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
at java.lang.Class.getConstructor0(Class.java:2763)
at java.lang.Class.getDeclaredConstructor(Class.java:2021)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
... 4 more
14/12/09 16:10:53 FATAL sharedcachemanager.SharedCacheManager: Error starting
SharedCacheManager
java.lang.RuntimeException: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156)
Caused by: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
at java.lang.Class.getConstructor0(Class.java:2763)
at java.lang.Class.getDeclaredConstructor(Class.java:2021)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
... 4 more
{noformat}
This JIRA is to add a 0-argument constructor to SCMStore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2944) SCMStore/InMemorySCMStore is not currently compatible with ReflectionUtils#newInstance
[
https://issues.apache.org/jira/browse/YARN-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Trezzo updated YARN-2944:
---
Attachment: (was: YARN-2944-trunk-v2.patch)
SCMStore/InMemorySCMStore is not currently compatible with
ReflectionUtils#newInstance
--
Key: YARN-2944
URL: https://issues.apache.org/jira/browse/YARN-2944
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Chris Trezzo
Assignee: Chris Trezzo
Priority: Minor
Attachments: YARN-2944-trunk-v1.patch, YARN-2944-trunk-v2.patch
Currently the Shared Cache Manager uses ReflectionUtils#newInstance to create
the SCMStore service. Unfortunately the SCMStore class does not have a
0-argument constructor.
On startup, the SCM fails with the following:
{noformat}
14/12/09 16:10:53 INFO service.AbstractService: Service SharedCacheManager
failed in state INITED; cause: java.lang.RuntimeException:
java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
java.lang.RuntimeException: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156)
Caused by: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
at java.lang.Class.getConstructor0(Class.java:2763)
at java.lang.Class.getDeclaredConstructor(Class.java:2021)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
... 4 more
14/12/09 16:10:53 FATAL sharedcachemanager.SharedCacheManager: Error starting
SharedCacheManager
java.lang.RuntimeException: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156)
Caused by: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
at java.lang.Class.getConstructor0(Class.java:2763)
at java.lang.Class.getDeclaredConstructor(Class.java:2021)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
... 4 more
{noformat}
This JIRA is to add a 0-argument constructor to SCMStore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2944) SCMStore/InMemorySCMStore is not currently compatible with ReflectionUtils#newInstance
[
https://issues.apache.org/jira/browse/YARN-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244813#comment-14244813
]
Hadoop QA commented on YARN-2944:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12686947/YARN-2944-trunk-v2.patch
against trunk revision 46612c7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6105//console
This message is automatically generated.
SCMStore/InMemorySCMStore is not currently compatible with
ReflectionUtils#newInstance
--
Key: YARN-2944
URL: https://issues.apache.org/jira/browse/YARN-2944
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Chris Trezzo
Assignee: Chris Trezzo
Priority: Minor
Attachments: YARN-2944-trunk-v1.patch, YARN-2944-trunk-v2.patch
Currently the Shared Cache Manager uses ReflectionUtils#newInstance to create
the SCMStore service. Unfortunately the SCMStore class does not have a
0-argument constructor.
On startup, the SCM fails with the following:
{noformat}
14/12/09 16:10:53 INFO service.AbstractService: Service SharedCacheManager
failed in state INITED; cause: java.lang.RuntimeException:
java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
java.lang.RuntimeException: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156)
Caused by: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
at java.lang.Class.getConstructor0(Class.java:2763)
at java.lang.Class.getDeclaredConstructor(Class.java:2021)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
... 4 more
14/12/09 16:10:53 FATAL sharedcachemanager.SharedCacheManager: Error starting
SharedCacheManager
java.lang.RuntimeException: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156)
Caused by: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
at java.lang.Class.getConstructor0(Class.java:2763)
at java.lang.Class.getDeclaredConstructor(Class.java:2021)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
... 4 more
{noformat}
This JIRA is to add a 0-argument constructor to SCMStore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2944) SCMStore/InMemorySCMStore is not currently compatible with ReflectionUtils#newInstance
[
https://issues.apache.org/jira/browse/YARN-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244840#comment-14244840
]
Hadoop QA commented on YARN-2944:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12686950/YARN-2944-trunk-v2.patch
against trunk revision 46612c7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6106//console
This message is automatically generated.
SCMStore/InMemorySCMStore is not currently compatible with
ReflectionUtils#newInstance
--
Key: YARN-2944
URL: https://issues.apache.org/jira/browse/YARN-2944
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Chris Trezzo
Assignee: Chris Trezzo
Priority: Minor
Attachments: YARN-2944-trunk-v1.patch, YARN-2944-trunk-v2.patch
Currently the Shared Cache Manager uses ReflectionUtils#newInstance to create
the SCMStore service. Unfortunately the SCMStore class does not have a
0-argument constructor.
On startup, the SCM fails with the following:
{noformat}
14/12/09 16:10:53 INFO service.AbstractService: Service SharedCacheManager
failed in state INITED; cause: java.lang.RuntimeException:
java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
java.lang.RuntimeException: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156)
Caused by: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
at java.lang.Class.getConstructor0(Class.java:2763)
at java.lang.Class.getDeclaredConstructor(Class.java:2021)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
... 4 more
14/12/09 16:10:53 FATAL sharedcachemanager.SharedCacheManager: Error starting
SharedCacheManager
java.lang.RuntimeException: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156)
Caused by: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
at java.lang.Class.getConstructor0(Class.java:2763)
at java.lang.Class.getDeclaredConstructor(Class.java:2021)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
... 4 more
{noformat}
This JIRA is to add a 0-argument constructor to SCMStore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2003) Support to process Job priority from Submission Context in AppAttemptAddedSchedulerEvent [RM side]
[ https://issues.apache.org/jira/browse/YARN-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244854#comment-14244854 ] Ashwin Shankar commented on YARN-2003: -- [~sunilg], Thanks for letting me know. Just wanted to emphasis that we are really interested in getting app priorities for fair scheduler committed soon, since we have few usecases in our production cluster that need this. Please let me know if you need help with the common code between the schedulers. Support to process Job priority from Submission Context in AppAttemptAddedSchedulerEvent [RM side] -- Key: YARN-2003 URL: https://issues.apache.org/jira/browse/YARN-2003 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Sunil G Assignee: Sunil G AppAttemptAddedSchedulerEvent should be able to receive the Job Priority from Submission Context and store. Later this can be used by Scheduler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2944) SCMStore/InMemorySCMStore is not currently compatible with ReflectionUtils#newInstance
[
https://issues.apache.org/jira/browse/YARN-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Trezzo updated YARN-2944:
---
Attachment: YARN-2944-trunk-v2.patch
Note: The InMemorySCMStore(AppChecker appChecker) constructor actually needs to
be public because it is used in TestSharedCacheUploaderService and
TestClientSCMProtocolService. I marked it visible for testing. I did reduce the
scope of the SCMStore constructor.
SCMStore/InMemorySCMStore is not currently compatible with
ReflectionUtils#newInstance
--
Key: YARN-2944
URL: https://issues.apache.org/jira/browse/YARN-2944
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Chris Trezzo
Assignee: Chris Trezzo
Priority: Minor
Attachments: YARN-2944-trunk-v1.patch, YARN-2944-trunk-v2.patch
Currently the Shared Cache Manager uses ReflectionUtils#newInstance to create
the SCMStore service. Unfortunately the SCMStore class does not have a
0-argument constructor.
On startup, the SCM fails with the following:
{noformat}
14/12/09 16:10:53 INFO service.AbstractService: Service SharedCacheManager
failed in state INITED; cause: java.lang.RuntimeException:
java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
java.lang.RuntimeException: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156)
Caused by: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
at java.lang.Class.getConstructor0(Class.java:2763)
at java.lang.Class.getDeclaredConstructor(Class.java:2021)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
... 4 more
14/12/09 16:10:53 FATAL sharedcachemanager.SharedCacheManager: Error starting
SharedCacheManager
java.lang.RuntimeException: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156)
Caused by: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
at java.lang.Class.getConstructor0(Class.java:2763)
at java.lang.Class.getDeclaredConstructor(Class.java:2021)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
... 4 more
{noformat}
This JIRA is to add a 0-argument constructor to SCMStore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2203) Web UI for cache manager
[ https://issues.apache.org/jira/browse/YARN-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Trezzo updated YARN-2203: --- Attachment: SCMUI-trunk-v4.png Attaching v4 screen shot of the overview page. Web UI for cache manager Key: YARN-2203 URL: https://issues.apache.org/jira/browse/YARN-2203 Project: Hadoop YARN Issue Type: Sub-task Reporter: Chris Trezzo Assignee: Chris Trezzo Attachments: SCMUI-trunk-v4.png, YARN-2203-trunk-v1.patch, YARN-2203-trunk-v2.patch, YARN-2203-trunk-v3.patch Implement the web server and web ui for the cache manager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2203) Web UI for cache manager
[ https://issues.apache.org/jira/browse/YARN-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Trezzo updated YARN-2203: --- Attachment: YARN-2203-trunk-v4.patch Attaching v4 of the patch. Web UI for cache manager Key: YARN-2203 URL: https://issues.apache.org/jira/browse/YARN-2203 Project: Hadoop YARN Issue Type: Sub-task Reporter: Chris Trezzo Assignee: Chris Trezzo Attachments: SCMUI-trunk-v4.png, YARN-2203-trunk-v1.patch, YARN-2203-trunk-v2.patch, YARN-2203-trunk-v3.patch, YARN-2203-trunk-v4.patch Implement the web server and web ui for the cache manager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2944) SCMStore/InMemorySCMStore is not currently compatible with ReflectionUtils#newInstance
[
https://issues.apache.org/jira/browse/YARN-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244941#comment-14244941
]
Hadoop QA commented on YARN-2944:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12686961/YARN-2944-trunk-v2.patch
against trunk revision 46612c7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1217 javac
compiler warnings (more than the trunk's current 155 warnings).
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
49 warning messages.
See
https://builds.apache.org/job/PreCommit-YARN-Build/6107//artifact/patchprocess/diffJavadocWarnings.txt
for details.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 25 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6107//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6107//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Javac warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6107//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6107//console
This message is automatically generated.
SCMStore/InMemorySCMStore is not currently compatible with
ReflectionUtils#newInstance
--
Key: YARN-2944
URL: https://issues.apache.org/jira/browse/YARN-2944
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Chris Trezzo
Assignee: Chris Trezzo
Priority: Minor
Attachments: YARN-2944-trunk-v1.patch, YARN-2944-trunk-v2.patch
Currently the Shared Cache Manager uses ReflectionUtils#newInstance to create
the SCMStore service. Unfortunately the SCMStore class does not have a
0-argument constructor.
On startup, the SCM fails with the following:
{noformat}
14/12/09 16:10:53 INFO service.AbstractService: Service SharedCacheManager
failed in state INITED; cause: java.lang.RuntimeException:
java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
java.lang.RuntimeException: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156)
Caused by: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
at java.lang.Class.getConstructor0(Class.java:2763)
at java.lang.Class.getDeclaredConstructor(Class.java:2021)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
... 4 more
14/12/09 16:10:53 FATAL sharedcachemanager.SharedCacheManager: Error starting
SharedCacheManager
java.lang.RuntimeException: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156)
Caused by: java.lang.NoSuchMethodException:
[jira] [Commented] (YARN-2203) Web UI for cache manager
[
https://issues.apache.org/jira/browse/YARN-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244946#comment-14244946
]
Hadoop QA commented on YARN-2203:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12686964/YARN-2203-trunk-v4.patch
against trunk revision 7784b10.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 eclipse:eclipse{color}. The patch failed to build with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6108//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6108//console
This message is automatically generated.
Web UI for cache manager
Key: YARN-2203
URL: https://issues.apache.org/jira/browse/YARN-2203
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Chris Trezzo
Assignee: Chris Trezzo
Attachments: SCMUI-trunk-v4.png, YARN-2203-trunk-v1.patch,
YARN-2203-trunk-v2.patch, YARN-2203-trunk-v3.patch, YARN-2203-trunk-v4.patch
Implement the web server and web ui for the cache manager.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2944) SCMStore/InMemorySCMStore is not currently compatible with ReflectionUtils#newInstance
[
https://issues.apache.org/jira/browse/YARN-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244953#comment-14244953
]
Chris Trezzo commented on YARN-2944:
Findbugs/javac/javadoc warnings seem to be erroneous/unrelated. The patch
should be good to go.
SCMStore/InMemorySCMStore is not currently compatible with
ReflectionUtils#newInstance
--
Key: YARN-2944
URL: https://issues.apache.org/jira/browse/YARN-2944
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Chris Trezzo
Assignee: Chris Trezzo
Priority: Minor
Attachments: YARN-2944-trunk-v1.patch, YARN-2944-trunk-v2.patch
Currently the Shared Cache Manager uses ReflectionUtils#newInstance to create
the SCMStore service. Unfortunately the SCMStore class does not have a
0-argument constructor.
On startup, the SCM fails with the following:
{noformat}
14/12/09 16:10:53 INFO service.AbstractService: Service SharedCacheManager
failed in state INITED; cause: java.lang.RuntimeException:
java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
java.lang.RuntimeException: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156)
Caused by: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
at java.lang.Class.getConstructor0(Class.java:2763)
at java.lang.Class.getDeclaredConstructor(Class.java:2021)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
... 4 more
14/12/09 16:10:53 FATAL sharedcachemanager.SharedCacheManager: Error starting
SharedCacheManager
java.lang.RuntimeException: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156)
Caused by: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
at java.lang.Class.getConstructor0(Class.java:2763)
at java.lang.Class.getDeclaredConstructor(Class.java:2021)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
... 4 more
{noformat}
This JIRA is to add a 0-argument constructor to SCMStore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2203) Web UI for cache manager
[ https://issues.apache.org/jira/browse/YARN-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244964#comment-14244964 ] Chris Trezzo commented on YARN-2203: eclipse:eclipse works fine locally. Also, this patch has been tested manually using a psudo-distributed cluster. Web UI for cache manager Key: YARN-2203 URL: https://issues.apache.org/jira/browse/YARN-2203 Project: Hadoop YARN Issue Type: Sub-task Reporter: Chris Trezzo Assignee: Chris Trezzo Attachments: SCMUI-trunk-v4.png, YARN-2203-trunk-v1.patch, YARN-2203-trunk-v2.patch, YARN-2203-trunk-v3.patch, YARN-2203-trunk-v4.patch Implement the web server and web ui for the cache manager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2944) SCMStore/InMemorySCMStore is not currently compatible with ReflectionUtils#newInstance
[
https://issues.apache.org/jira/browse/YARN-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244978#comment-14244978
]
Sangjin Lee commented on YARN-2944:
---
LGTM. Thanks [~ctrezzo]!
SCMStore/InMemorySCMStore is not currently compatible with
ReflectionUtils#newInstance
--
Key: YARN-2944
URL: https://issues.apache.org/jira/browse/YARN-2944
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Chris Trezzo
Assignee: Chris Trezzo
Priority: Minor
Attachments: YARN-2944-trunk-v1.patch, YARN-2944-trunk-v2.patch
Currently the Shared Cache Manager uses ReflectionUtils#newInstance to create
the SCMStore service. Unfortunately the SCMStore class does not have a
0-argument constructor.
On startup, the SCM fails with the following:
{noformat}
14/12/09 16:10:53 INFO service.AbstractService: Service SharedCacheManager
failed in state INITED; cause: java.lang.RuntimeException:
java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
java.lang.RuntimeException: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156)
Caused by: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
at java.lang.Class.getConstructor0(Class.java:2763)
at java.lang.Class.getDeclaredConstructor(Class.java:2021)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
... 4 more
14/12/09 16:10:53 FATAL sharedcachemanager.SharedCacheManager: Error starting
SharedCacheManager
java.lang.RuntimeException: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.createSCMStoreService(SharedCacheManager.java:103)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.serviceInit(SharedCacheManager.java:65)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager.main(SharedCacheManager.java:156)
Caused by: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore.init()
at java.lang.Class.getConstructor0(Class.java:2763)
at java.lang.Class.getDeclaredConstructor(Class.java:2021)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
... 4 more
{noformat}
This JIRA is to add a 0-argument constructor to SCMStore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2356) yarn status command for non-existent application/application attempt/container is too verbose
[
https://issues.apache.org/jira/browse/YARN-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245019#comment-14245019
]
Hadoop QA commented on YARN-2356:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12686822/0004-YARN-2356.patch
against trunk revision c78e3a7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 10 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client:
org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6109//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6109//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-client.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6109//console
This message is automatically generated.
yarn status command for non-existent application/application
attempt/container is too verbose
--
Key: YARN-2356
URL: https://issues.apache.org/jira/browse/YARN-2356
Project: Hadoop YARN
Issue Type: Bug
Components: client
Reporter: Sunil G
Assignee: Sunil G
Priority: Minor
Attachments: 0001-YARN-2356.patch, 0002-YARN-2356.patch,
0003-YARN-2356.patch, 0004-YARN-2356.patch, Yarn-2356.1.patch
*yarn application -status* or *applicationattempt -status* or *container
status* commands can suppress exception such as ApplicationNotFound,
ApplicationAttemptNotFound and ContainerNotFound for non-existent entries in
RM or History Server.
For example, below exception can be suppressed better
sunildev@host-a:~/hadoop/hadoop/bin ./yarn application -status
application_1402668848165_0015
No GC_PROFILE is given. Defaults to medium.
14/07/25 16:21:45 INFO client.RMProxy: Connecting to ResourceManager at
/10.18.40.77:45022
Exception in thread main
org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application
with id 'application_1402668848165_0015' doesn't exist in RM.
at
org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:285)
at
org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:145)
at
org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:321)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:607)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:932)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2099)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2095)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1626)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2093)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
at
org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:101)
at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:166)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[jira] [Created] (YARN-2959) Fair Scheduler fifo option can violate FIFO behavior and cause deadlock among jobs
Ashwin Shankar created YARN-2959: Summary: Fair Scheduler fifo option can violate FIFO behavior and cause deadlock among jobs Key: YARN-2959 URL: https://issues.apache.org/jira/browse/YARN-2959 Project: Hadoop YARN Issue Type: Bug Components: fairscheduler Reporter: Ashwin Shankar We have a cluster which run jobs in fifo order(due to the nature of those jobs) using Fair scheduler's fifo option. Recently we found jobs deadlocked in the cluster, here is what happened : There were two jobs,say A and B. A was submitted before B. Both were in PENDING state since the cluster was busy. When containers freed up, the two pending jobs got their AM containers at about the same time. However Job B's AM or appattempt1 registered with RM a little earlier than Job A and grabbed available containers at that time, and satisfied a fraction of its requirement. Note, JobB can't make progress until it gets all its requirement satisfied. Next, JobA's appattempt1 registered with RM and since JobA was submitted earlier, RM stops allocating containers to JobB and starts allocating to JobA, satisfying a fraction of its requirement as well. Now together jobA,jobB hold the entire cluster, but neither can progress and are deadlocked since their resource requests are partially satisfied. Note:Above is an example with 2 jobs, however the deadlock can happen with n jobs : J1..Jn if the sequence of AM registration is Jn, J(n-1),..J1. Solution : one proposed solution is to order the fifo queue by appattempt start/register time instead of app submit time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2958) RMStateStore seems to unnecessarily and wronly store sequence number separately
Zhijie Shen created YARN-2958:
-
Summary: RMStateStore seems to unnecessarily and wronly store
sequence number separately
Key: YARN-2958
URL: https://issues.apache.org/jira/browse/YARN-2958
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Reporter: Zhijie Shen
It seems that RMStateStore updates last sequence number when storing or
updating each individual DT, to recover the latest sequence number when RM
restarting.
First, the current logic seems to be problematic:
{code}
public synchronized void updateRMDelegationTokenAndSequenceNumber(
RMDelegationTokenIdentifier rmDTIdentifier, Long renewDate,
int latestSequenceNumber) {
if(isFencedState()) {
LOG.info(State store is in Fenced state. Can't update RM Delegation
Token.);
return;
}
try {
updateRMDelegationTokenAndSequenceNumberInternal(rmDTIdentifier,
renewDate,
latestSequenceNumber);
} catch (Exception e) {
notifyStoreOperationFailed(e);
}
}
{code}
{code}
@Override
protected void updateStoredToken(RMDelegationTokenIdentifier id,
long renewDate) {
try {
LOG.info(updating RMDelegation token with sequence number:
+ id.getSequenceNumber());
rmContext.getStateStore().updateRMDelegationTokenAndSequenceNumber(id,
renewDate, id.getSequenceNumber());
} catch (Exception e) {
LOG.error(Error in updating persisted RMDelegationToken with sequence
number:
+ id.getSequenceNumber());
ExitUtil.terminate(1, e);
}
}
{code}
According to code above, even when renewing a DT, the last sequence number is
updated in the store, which is wrong. For example, we have the following
sequence:
1. Get DT 1 (seq = 1)
2. Get DT 2( seq = 2)
3. Renew DT 1 (seq = 1)
4. Restart RM
The stored and then recovered last sequence number is 1. It makes the next
created DT after RM restarting will conflict with DT 2 on sequence num.
Second, the aforementioned bug doesn't happen actually, because the recovered
last sequence num has been overwritten at by the correctly one.
{code}
public void recover(RMState rmState) throws Exception {
LOG.info(recovering RMDelegationTokenSecretManager.);
// recover RMDTMasterKeys
for (DelegationKey dtKey : rmState.getRMDTSecretManagerState()
.getMasterKeyState()) {
addKey(dtKey);
}
// recover RMDelegationTokens
MapRMDelegationTokenIdentifier, Long rmDelegationTokens =
rmState.getRMDTSecretManagerState().getTokenState();
this.delegationTokenSequenceNumber =
rmState.getRMDTSecretManagerState().getDTSequenceNumber();
for (Map.EntryRMDelegationTokenIdentifier, Long entry : rmDelegationTokens
.entrySet()) {
addPersistedDelegationToken(entry.getKey(), entry.getValue());
}
}
{code}
The code above recovers delegationTokenSequenceNumber by reading the last
sequence number in the store. It could be wrong. Fortunately,
delegationTokenSequenceNumber updates it to the right number.
{code}
if (identifier.getSequenceNumber() getDelegationTokenSeqNum()) {
setDelegationTokenSeqNum(identifier.getSequenceNumber());
}
{code}
All the stored identifiers will be gone through, and
delegationTokenSequenceNumber will be set to the largest sequence number among
these identifiers. Therefore, new DT will be assigned a sequence number which
is always larger than that of all the recovered DT.
To sum up, two negatives make a positive, but it's good to fix the issue.
Please let me know if I've missed something here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[ https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245035#comment-14245035 ] Zhijie Shen commented on YARN-2837: --- bq. we need to keep track of the latest sequenceNumber so that it can be recovered. For the reason of YARN-2958, I don't think it's proper to following the way of RMStateStore. Please let me know if I missed something. Timeline server needs to recover the timeline DT when restarting Key: YARN-2837 URL: https://issues.apache.org/jira/browse/YARN-2837 Project: Hadoop YARN Issue Type: New Feature Components: timelineserver Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Blocker Fix For: 2.7.0 Attachments: YARN-2837.1.patch, YARN-2837.2.patch, YARN-2837.3.patch, YARN-2837.4.patch, YARN-2837.5.patch, YARN-2837.6.patch Timeline server needs to recover the stateful information when restarting as RM/NM/JHS does now. So far the stateful information only includes the timeline DT. Without recovery, the timeline DT of the existing YARN apps is not long valid, and cannot be renewed any more after the timeline server is restarted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (YARN-2654) Revisit all shared cache config parameters to ensure quality names
[ https://issues.apache.org/jira/browse/YARN-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Trezzo reassigned YARN-2654: -- Assignee: Chris Trezzo Revisit all shared cache config parameters to ensure quality names -- Key: YARN-2654 URL: https://issues.apache.org/jira/browse/YARN-2654 Project: Hadoop YARN Issue Type: Sub-task Reporter: Chris Trezzo Assignee: Chris Trezzo Priority: Blocker Revisit all the shared cache config parameters in YarnConfiguration and yarn-default.xml to ensure quality names. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2960) Add documentation for the YARN shared cache
Chris Trezzo created YARN-2960: -- Summary: Add documentation for the YARN shared cache Key: YARN-2960 URL: https://issues.apache.org/jira/browse/YARN-2960 Project: Hadoop YARN Issue Type: Sub-task Reporter: Chris Trezzo Add documentation around the architecture, api's and administration of the YARN shared cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2950) Change message to mandate, not suggest JS requirement on UI
[
https://issues.apache.org/jira/browse/YARN-2950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated YARN-2950:
--
Attachment: YARN-2950-2.patch
Thanks Dustin!
Looks good to me. I've gone ahead and made a small change to keep the line
lengths less than 80 characters as per formatting requirements.
Committing in a bit.
Change message to mandate, not suggest JS requirement on UI
---
Key: YARN-2950
URL: https://issues.apache.org/jira/browse/YARN-2950
Project: Hadoop YARN
Issue Type: Improvement
Components: webapp
Affects Versions: 2.5.0
Reporter: Harsh J
Assignee: Dustin Cote
Priority: Minor
Labels: newbie
Attachments: YARN-2950-1.patch, YARN-2950-2.patch
Most of YARN's UIs do not work with JavaScript disabled on the browser, cause
they appear to send back data as JS arrays instead of within the actual HTML
content.
The JQueryUI prints only a mild warning about this suggesting that {{This
page works best with javascript enabled.}}, when in fact it ought to be
{{This page will not function without javascript enabled. Please enable
javascript on your browser.}} or something as such (more direct).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (YARN-2958) RMStateStore seems to unnecessarily and wronly store sequence number separately
[
https://issues.apache.org/jira/browse/YARN-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena reassigned YARN-2958:
--
Assignee: Varun Saxena
RMStateStore seems to unnecessarily and wronly store sequence number
separately
---
Key: YARN-2958
URL: https://issues.apache.org/jira/browse/YARN-2958
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Reporter: Zhijie Shen
Assignee: Varun Saxena
It seems that RMStateStore updates last sequence number when storing or
updating each individual DT, to recover the latest sequence number when RM
restarting.
First, the current logic seems to be problematic:
{code}
public synchronized void updateRMDelegationTokenAndSequenceNumber(
RMDelegationTokenIdentifier rmDTIdentifier, Long renewDate,
int latestSequenceNumber) {
if(isFencedState()) {
LOG.info(State store is in Fenced state. Can't update RM Delegation
Token.);
return;
}
try {
updateRMDelegationTokenAndSequenceNumberInternal(rmDTIdentifier,
renewDate,
latestSequenceNumber);
} catch (Exception e) {
notifyStoreOperationFailed(e);
}
}
{code}
{code}
@Override
protected void updateStoredToken(RMDelegationTokenIdentifier id,
long renewDate) {
try {
LOG.info(updating RMDelegation token with sequence number:
+ id.getSequenceNumber());
rmContext.getStateStore().updateRMDelegationTokenAndSequenceNumber(id,
renewDate, id.getSequenceNumber());
} catch (Exception e) {
LOG.error(Error in updating persisted RMDelegationToken with sequence
number:
+ id.getSequenceNumber());
ExitUtil.terminate(1, e);
}
}
{code}
According to code above, even when renewing a DT, the last sequence number is
updated in the store, which is wrong. For example, we have the following
sequence:
1. Get DT 1 (seq = 1)
2. Get DT 2( seq = 2)
3. Renew DT 1 (seq = 1)
4. Restart RM
The stored and then recovered last sequence number is 1. It makes the next
created DT after RM restarting will conflict with DT 2 on sequence num.
Second, the aforementioned bug doesn't happen actually, because the recovered
last sequence num has been overwritten at by the correctly one.
{code}
public void recover(RMState rmState) throws Exception {
LOG.info(recovering RMDelegationTokenSecretManager.);
// recover RMDTMasterKeys
for (DelegationKey dtKey : rmState.getRMDTSecretManagerState()
.getMasterKeyState()) {
addKey(dtKey);
}
// recover RMDelegationTokens
MapRMDelegationTokenIdentifier, Long rmDelegationTokens =
rmState.getRMDTSecretManagerState().getTokenState();
this.delegationTokenSequenceNumber =
rmState.getRMDTSecretManagerState().getDTSequenceNumber();
for (Map.EntryRMDelegationTokenIdentifier, Long entry :
rmDelegationTokens
.entrySet()) {
addPersistedDelegationToken(entry.getKey(), entry.getValue());
}
}
{code}
The code above recovers delegationTokenSequenceNumber by reading the last
sequence number in the store. It could be wrong. Fortunately,
delegationTokenSequenceNumber updates it to the right number.
{code}
if (identifier.getSequenceNumber() getDelegationTokenSeqNum()) {
setDelegationTokenSeqNum(identifier.getSequenceNumber());
}
{code}
All the stored identifiers will be gone through, and
delegationTokenSequenceNumber will be set to the largest sequence number
among these identifiers. Therefore, new DT will be assigned a sequence number
which is always larger than that of all the recovered DT.
To sum up, two negatives make a positive, but it's good to fix the issue.
Please let me know if I've missed something here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2950) Change message to mandate, not suggest JS requirement on UI
[
https://issues.apache.org/jira/browse/YARN-2950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245101#comment-14245101
]
Hudson commented on YARN-2950:
--
FAILURE: Integrated in Hadoop-trunk-Commit #6712 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/6712/])
YARN-2950. Change message to mandate, not suggest JS requirement on UI.
Contributed by Dustin Cote. (harsh: rev
0e37bbc8e3f8e96acd96522face2f4bb01584cb4)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/view/JQueryUI.java
* hadoop-yarn-project/CHANGES.txt
Change message to mandate, not suggest JS requirement on UI
---
Key: YARN-2950
URL: https://issues.apache.org/jira/browse/YARN-2950
Project: Hadoop YARN
Issue Type: Improvement
Components: webapp
Affects Versions: 2.5.0
Reporter: Harsh J
Assignee: Dustin Cote
Priority: Minor
Labels: newbie
Fix For: 2.7.0
Attachments: YARN-2950-1.patch, YARN-2950-2.patch
Most of YARN's UIs do not work with JavaScript disabled on the browser, cause
they appear to send back data as JS arrays instead of within the actual HTML
content.
The JQueryUI prints only a mild warning about this suggesting that {{This
page works best with javascript enabled.}}, when in fact it ought to be
{{This page will not function without javascript enabled. Please enable
javascript on your browser.}} or something as such (more direct).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2937) Fix new findbugs warnings in hadoop-yarn-nodemanager
[
https://issues.apache.org/jira/browse/YARN-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245160#comment-14245160
]
Varun Saxena commented on YARN-2937:
[~zjshen],
bq. What's the problem around Random?
There is a problem with Maths#abs.
{{Maths.abs(Long.MIN_VALUE)==Long.MIN_VALUE}} i.e. Maths.abs doesnt work with
lowest value of long. But Random may return that value. So changed Random to
RandomUtils to give only positive values. Maths.abs can probably be removed in
this case. Random has {{getInt}} method which can return positive values but no
corresponding method for nextLong
bq. Can we use IOUtils?
Yes, will use it.
bq. Is this removed because bufReader will consequently close fileReader too?
No. This has been removed because instance of class {{FileReader}} has been
removed and replaced with {{FileInputStream}} + {{InputStreamReader}}. This is
because FileReader doesnt allow us to set Charset encoding. And yes,
BufferedReader will close the underlying FileReader and FileReader will close
the underlying stream.
bq. Again, would you mind explaining a bit about what have been excluded in
findbugs-exclude.xml?
The exclusions are corresponding to *RV_RETURN_VALUE_IGNORED_BAD_PRACTICE*.
Findbugs was complaining about return value of {{ThreadPoolExecutor#submit}}
being ignored. This method returns a {{FutureT}} object. Generally
{{Future#get}} is called to check the return value of executor. But this method
is blocking and waits for thread execution to complete. If I call this, it
would break the current behavior. Tried it as well and some test cases failed.
Another alternative is to call {{Future#isDone}} but this most likely will
always return false if you check it immediately. So I see no point in checking
return value of call to submit. Hence, added it in exclusions. Findbugs was
raising issue for below 2 lines.
{code:title=ContainersLauncher.java:118|borderStyle=solid}
containerLauncher.submit(launch);
{code}
{code:title=SharedCacheUploadService.java:118|borderStyle=solid}
uploaderPool.submit(uploader);
{code}
Fix new findbugs warnings in hadoop-yarn-nodemanager
Key: YARN-2937
URL: https://issues.apache.org/jira/browse/YARN-2937
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Varun Saxena
Assignee: Varun Saxena
Fix For: 2.7.0
Attachments: HADOOP-11373.patch, YARN-2937.001.patch,
YARN-2937.002.patch, YARN-2937.003.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2356) yarn status command for non-existent application/application attempt/container is too verbose
[ https://issues.apache.org/jira/browse/YARN-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245202#comment-14245202 ] Sunil G commented on YARN-2356: --- Hi Devaraj These findbugs problems are not induced by current JIRA. There were some rule changes done for java 7. I feel that caused the same. I cud open a new JIRA to fix this warning of PrintStream writer in all CLI classes. Kindly advice. Also test case failure is not related. yarn status command for non-existent application/application attempt/container is too verbose -- Key: YARN-2356 URL: https://issues.apache.org/jira/browse/YARN-2356 Project: Hadoop YARN Issue Type: Bug Components: client Reporter: Sunil G Assignee: Sunil G Priority: Minor Attachments: 0001-YARN-2356.patch, 0002-YARN-2356.patch, 0003-YARN-2356.patch, 0004-YARN-2356.patch, Yarn-2356.1.patch *yarn application -status* or *applicationattempt -status* or *container status* commands can suppress exception such as ApplicationNotFound, ApplicationAttemptNotFound and ContainerNotFound for non-existent entries in RM or History Server. For example, below exception can be suppressed better sunildev@host-a:~/hadoop/hadoop/bin ./yarn application -status application_1402668848165_0015 No GC_PROFILE is given. Defaults to medium. 14/07/25 16:21:45 INFO client.RMProxy: Connecting to ResourceManager at /10.18.40.77:45022 Exception in thread main org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application with id 'application_1402668848165_0015' doesn't exist in RM. at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:285) at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:145) at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:321) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:607) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:932) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2099) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2095) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1626) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2093) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:101) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:166) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103) at $Proxy12.getApplicationReport(Unknown Source) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:291) at org.apache.hadoop.yarn.client.cli.ApplicationCLI.printApplicationReport(ApplicationCLI.java:428) at org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:153) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.hadoop.yarn.client.cli.ApplicationCLI.main(ApplicationCLI.java:76) Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException): Application with id 'application_1402668848165_0015' doesn't exist in RM. -- This message was
[jira] [Commented] (YARN-2356) yarn status command for non-existent application/application attempt/container is too verbose
[ https://issues.apache.org/jira/browse/YARN-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245221#comment-14245221 ] Varun Saxena commented on YARN-2356: Yes [~sunilg] is correct. These findbugs will be handled by YARN-2940 yarn status command for non-existent application/application attempt/container is too verbose -- Key: YARN-2356 URL: https://issues.apache.org/jira/browse/YARN-2356 Project: Hadoop YARN Issue Type: Bug Components: client Reporter: Sunil G Assignee: Sunil G Priority: Minor Attachments: 0001-YARN-2356.patch, 0002-YARN-2356.patch, 0003-YARN-2356.patch, 0004-YARN-2356.patch, Yarn-2356.1.patch *yarn application -status* or *applicationattempt -status* or *container status* commands can suppress exception such as ApplicationNotFound, ApplicationAttemptNotFound and ContainerNotFound for non-existent entries in RM or History Server. For example, below exception can be suppressed better sunildev@host-a:~/hadoop/hadoop/bin ./yarn application -status application_1402668848165_0015 No GC_PROFILE is given. Defaults to medium. 14/07/25 16:21:45 INFO client.RMProxy: Connecting to ResourceManager at /10.18.40.77:45022 Exception in thread main org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application with id 'application_1402668848165_0015' doesn't exist in RM. at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:285) at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:145) at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:321) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:607) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:932) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2099) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2095) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1626) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2093) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:101) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:166) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103) at $Proxy12.getApplicationReport(Unknown Source) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:291) at org.apache.hadoop.yarn.client.cli.ApplicationCLI.printApplicationReport(ApplicationCLI.java:428) at org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:153) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.hadoop.yarn.client.cli.ApplicationCLI.main(ApplicationCLI.java:76) Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException): Application with id 'application_1402668848165_0015' doesn't exist in RM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
