[jira] [Created] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
Zhijie Shen created YARN-2837: - Summary: Timeline server needs to recover the timeline DT when restarting Key: YARN-2837 URL: https://issues.apache.org/jira/browse/YARN-2837 Project: Hadoop YARN Issue Type: New Feature Components: timelineserver Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Blocker Timeline server needs to recover the stateful information when restarting as RM/NM/JHS does now. So far the stateful information only includes the timeline DT. Without recovery, the timeline DT of the existing YARN apps is not long valid, and cannot be renewed any more after the timeline server is restarted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2836) RM behaviour on token renewal failures is broken
[ https://issues.apache.org/jira/browse/YARN-2836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204289#comment-14204289 ] Vinod Kumar Vavilapalli commented on YARN-2836: --- Linking related tickets. > RM behaviour on token renewal failures is broken > > > Key: YARN-2836 > URL: https://issues.apache.org/jira/browse/YARN-2836 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > > Found this while reviewing YARN-2834. > We now completely ignore token renewal failures. For things like Timeline > tokens which are automatically obtained whether the app needs it or not (we > should fix this to be user driven), we can ignore failures. But for HDFS > Tokens etc, ignoring failures is bad because it (1) wastes resources as AMs > will continue and eventually fail (2) app doesn't know what happened it fails > eventually. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2834) Resource manager crashed with Null Pointer Exception
[
https://issues.apache.org/jira/browse/YARN-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204290#comment-14204290
]
Vinod Kumar Vavilapalli commented on YARN-2834:
---
Filed YARN-2836 as a 2.7 blocker.
> Resource manager crashed with Null Pointer Exception
>
>
> Key: YARN-2834
> URL: https://issues.apache.org/jira/browse/YARN-2834
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yesha Vora
>Assignee: Jian He
>Priority: Blocker
> Fix For: 2.6.0
>
> Attachments: YARN-2834.1.patch
>
>
> Resource manager failed after restart.
> {noformat}
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initializeQueues(467)) - Initialized root queue root:
> numChildQueue= 2, capacity=1.0, absoluteCapacity=1.0,
> usedResources=usedCapacity=0.0, numApps=0, numContainers=0
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initializeQueueMappings(436)) - Initialized queue
> mappings, override: false
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initScheduler(305)) - Initialized CapacityScheduler
> with calculator=class
> org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator,
> minimumAllocation=<>, maximumAllocation=< vCores:32>>, asynchronousScheduling=false, asyncScheduleInterval=5ms
> 2014-11-09 04:12:53,015 INFO service.AbstractService
> (AbstractService.java:noteFailure(272)) - Service ResourceManager failed in
> state STARTED; cause: java.lang.NullPointerException
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.addApplicationAttempt(CapacityScheduler.java:734)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1089)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AttemptRecoveredTransition.transition(RMAppAttemptImpl.java:1041)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AttemptRecoveredTransition.transition(RMAppAttemptImpl.java:1005)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:757)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:106)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.recoverAppAttempts(RMAppImpl.java:821)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.access$1900(RMAppImpl.java:101)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:843)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:826)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:701)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:312)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:413)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1207)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:590)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(Resource
[jira] [Created] (YARN-2836) RM behaviour on token renewal failures is broken
Vinod Kumar Vavilapalli created YARN-2836: - Summary: RM behaviour on token renewal failures is broken Key: YARN-2836 URL: https://issues.apache.org/jira/browse/YARN-2836 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Priority: Blocker Found this while reviewing YARN-2834. We now completely ignore token renewal failures. For things like Timeline tokens which are automatically obtained whether the app needs it or not (we should fix this to be user driven), we can ignore failures. But for HDFS Tokens etc, ignoring failures is bad because it (1) wastes resources as AMs will continue and eventually fail (2) app doesn't know what happened it fails eventually. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2834) Resource manager crashed with Null Pointer Exception
[
https://issues.apache.org/jira/browse/YARN-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204269#comment-14204269
]
Hudson commented on YARN-2834:
--
FAILURE: Integrated in Hadoop-trunk-Commit #6501 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/6501/])
YARN-2834. Fixed ResourceManager to ignore token-renewal failures on recovery
consistent with the (somewhat incorrect) behaviour in the non-recovery case.
Contributed by Jian He. (vinodkv: rev e76faebc9589654e83c8244ef9aff88391e56b80)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestWorkPreservingRMRestart.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
> Resource manager crashed with Null Pointer Exception
>
>
> Key: YARN-2834
> URL: https://issues.apache.org/jira/browse/YARN-2834
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yesha Vora
>Assignee: Jian He
>Priority: Blocker
> Attachments: YARN-2834.1.patch
>
>
> Resource manager failed after restart.
> {noformat}
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initializeQueues(467)) - Initialized root queue root:
> numChildQueue= 2, capacity=1.0, absoluteCapacity=1.0,
> usedResources=usedCapacity=0.0, numApps=0, numContainers=0
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initializeQueueMappings(436)) - Initialized queue
> mappings, override: false
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initScheduler(305)) - Initialized CapacityScheduler
> with calculator=class
> org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator,
> minimumAllocation=<>, maximumAllocation=< vCores:32>>, asynchronousScheduling=false, asyncScheduleInterval=5ms
> 2014-11-09 04:12:53,015 INFO service.AbstractService
> (AbstractService.java:noteFailure(272)) - Service ResourceManager failed in
> state STARTED; cause: java.lang.NullPointerException
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.addApplicationAttempt(CapacityScheduler.java:734)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1089)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AttemptRecoveredTransition.transition(RMAppAttemptImpl.java:1041)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AttemptRecoveredTransition.transition(RMAppAttemptImpl.java:1005)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:757)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:106)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.recoverAppAttempts(RMAppImpl.java:821)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.access$1900(RMAppImpl.java:101)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:843)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:826)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
[jira] [Commented] (YARN-2834) Resource manager crashed with Null Pointer Exception
[
https://issues.apache.org/jira/browse/YARN-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204266#comment-14204266
]
Vinod Kumar Vavilapalli commented on YARN-2834:
---
This looks good, +1. Checking this in.
> Resource manager crashed with Null Pointer Exception
>
>
> Key: YARN-2834
> URL: https://issues.apache.org/jira/browse/YARN-2834
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yesha Vora
>Assignee: Jian He
>Priority: Blocker
> Attachments: YARN-2834.1.patch
>
>
> Resource manager failed after restart.
> {noformat}
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initializeQueues(467)) - Initialized root queue root:
> numChildQueue= 2, capacity=1.0, absoluteCapacity=1.0,
> usedResources=usedCapacity=0.0, numApps=0, numContainers=0
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initializeQueueMappings(436)) - Initialized queue
> mappings, override: false
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initScheduler(305)) - Initialized CapacityScheduler
> with calculator=class
> org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator,
> minimumAllocation=<>, maximumAllocation=< vCores:32>>, asynchronousScheduling=false, asyncScheduleInterval=5ms
> 2014-11-09 04:12:53,015 INFO service.AbstractService
> (AbstractService.java:noteFailure(272)) - Service ResourceManager failed in
> state STARTED; cause: java.lang.NullPointerException
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.addApplicationAttempt(CapacityScheduler.java:734)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1089)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AttemptRecoveredTransition.transition(RMAppAttemptImpl.java:1041)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AttemptRecoveredTransition.transition(RMAppAttemptImpl.java:1005)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:757)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:106)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.recoverAppAttempts(RMAppImpl.java:821)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.access$1900(RMAppImpl.java:101)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:843)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:826)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:701)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:312)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:413)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1207)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:590)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:1014)
>
[jira] [Updated] (YARN-2834) Resource manager crashed with Null Pointer Exception
[
https://issues.apache.org/jira/browse/YARN-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated YARN-2834:
Priority: Blocker (was: Critical)
> Resource manager crashed with Null Pointer Exception
>
>
> Key: YARN-2834
> URL: https://issues.apache.org/jira/browse/YARN-2834
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yesha Vora
>Assignee: Jian He
>Priority: Blocker
> Attachments: YARN-2834.1.patch
>
>
> Resource manager failed after restart.
> {noformat}
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initializeQueues(467)) - Initialized root queue root:
> numChildQueue= 2, capacity=1.0, absoluteCapacity=1.0,
> usedResources=usedCapacity=0.0, numApps=0, numContainers=0
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initializeQueueMappings(436)) - Initialized queue
> mappings, override: false
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initScheduler(305)) - Initialized CapacityScheduler
> with calculator=class
> org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator,
> minimumAllocation=<>, maximumAllocation=< vCores:32>>, asynchronousScheduling=false, asyncScheduleInterval=5ms
> 2014-11-09 04:12:53,015 INFO service.AbstractService
> (AbstractService.java:noteFailure(272)) - Service ResourceManager failed in
> state STARTED; cause: java.lang.NullPointerException
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.addApplicationAttempt(CapacityScheduler.java:734)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1089)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AttemptRecoveredTransition.transition(RMAppAttemptImpl.java:1041)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AttemptRecoveredTransition.transition(RMAppAttemptImpl.java:1005)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:757)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:106)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.recoverAppAttempts(RMAppImpl.java:821)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.access$1900(RMAppImpl.java:101)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:843)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:826)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:701)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:312)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:413)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1207)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:590)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:1014)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(R
[jira] [Commented] (YARN-2776) In HA mode, can't set ip but hostname to "yarn.resourcemanager.webapp.address.*"
[ https://issues.apache.org/jira/browse/YARN-2776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204208#comment-14204208 ] meiyoula commented on YARN-2776: [~aw]What do you think about this bug, why nobody attention it? > In HA mode, can't set ip but hostname to > "yarn.resourcemanager.webapp.address.*" > - > > Key: YARN-2776 > URL: https://issues.apache.org/jira/browse/YARN-2776 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: meiyoula > Attachments: YARN-2776.1.path > > > In HA mode, when setting yarn.resourcemanager.webapp.address.* with ip:port, > I run a spark application on yarn. The sparkui in yarn webui of 8080 is ok, > but spark own page has a bug when the address turns to yarn address. > But when setting yarn.resourcemanager.webapp.address.* with hostname:port, > the webs both ok. > The error in web: > HTTP ERROR: 500 > Problem accessing /stages/. Reason: > Server Error > The exception in log : > WARN | [qtp542345580-71] | /stages/ | > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:561) > javax.servlet.ServletException: Could not determine the proxy server for > redirection > at > org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.findRedirectUrl(AmIpFilter.java:183) > at > org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:139) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) > at org.eclipse.jetty.server.Server.handle(Server.java:370) > at > org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) > at > org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971) > at > org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033) > at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) > at > org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) > at > org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) > at > org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667) > at > org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2830) Add backwords compatible ContainerId.newInstance constructor for use within Tez Local Mode
[ https://issues.apache.org/jira/browse/YARN-2830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204171#comment-14204171 ] Hudson commented on YARN-2830: -- FAILURE: Integrated in Hadoop-trunk-Commit #6495 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6495/]) YARN-2830. Add backwords compatible ContainerId.newInstance constructor. Contributed by Jonathan Eagles. (acmurthy: rev 43cd07b408c6613d2c9aa89203cfa3110d830538) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRAppBenchmark.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestContainerResourceDecrease.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestMemoryApplicationHistoryStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/ProtocolHATestBase.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/TestContainerManagerSecurity.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/TestAHSWebApp.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestContainerResourceIncrease.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestContainerLaunchRPC.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAHSClient.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogsBlock.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestFileSystemApplicationHistoryStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/recovery/TestNMLeveldbStateStoreService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestWorkPreservingRMRestart.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ContainerId.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestYarnServerApiClasses.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestEventFlow.java * hadoop-
[jira] [Commented] (YARN-2834) Resource manager crashed with Null Pointer Exception
[
https://issues.apache.org/jira/browse/YARN-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204154#comment-14204154
]
Hadoop QA commented on YARN-2834:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12680505/YARN-2834.1.patch
against trunk revision 9ba8d8c.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5807//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5807//console
This message is automatically generated.
> Resource manager crashed with Null Pointer Exception
>
>
> Key: YARN-2834
> URL: https://issues.apache.org/jira/browse/YARN-2834
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yesha Vora
>Assignee: Jian He
>Priority: Critical
> Attachments: YARN-2834.1.patch
>
>
> Resource manager failed after restart.
> {noformat}
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initializeQueues(467)) - Initialized root queue root:
> numChildQueue= 2, capacity=1.0, absoluteCapacity=1.0,
> usedResources=usedCapacity=0.0, numApps=0, numContainers=0
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initializeQueueMappings(436)) - Initialized queue
> mappings, override: false
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initScheduler(305)) - Initialized CapacityScheduler
> with calculator=class
> org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator,
> minimumAllocation=<>, maximumAllocation=< vCores:32>>, asynchronousScheduling=false, asyncScheduleInterval=5ms
> 2014-11-09 04:12:53,015 INFO service.AbstractService
> (AbstractService.java:noteFailure(272)) - Service ResourceManager failed in
> state STARTED; cause: java.lang.NullPointerException
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.addApplicationAttempt(CapacityScheduler.java:734)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1089)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AttemptRecoveredTransition.transition(RMAppAttemptImpl.java:1041)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AttemptRecoveredTransition.transition(RMAppAttemptImpl.java:1005)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:757)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:106)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.recoverAppAttempts(RMAppImpl.java:821)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.access$1900(RMAppImpl.java:101)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:843)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.tran
[jira] [Commented] (YARN-2834) Resource manager crashed with Null Pointer Exception
[
https://issues.apache.org/jira/browse/YARN-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204132#comment-14204132
]
Zhijie Shen commented on YARN-2834:
---
bq. Anyways, treating renewal failures is broken today. I am okay ignoring
renewal failures during recovery in this ticket. But let's file a blocker for
handling them correctly in 2.7.
Thanks for your comments. +1 for this proposal.
> Resource manager crashed with Null Pointer Exception
>
>
> Key: YARN-2834
> URL: https://issues.apache.org/jira/browse/YARN-2834
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yesha Vora
>Assignee: Jian He
>Priority: Critical
> Attachments: YARN-2834.1.patch
>
>
> Resource manager failed after restart.
> {noformat}
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initializeQueues(467)) - Initialized root queue root:
> numChildQueue= 2, capacity=1.0, absoluteCapacity=1.0,
> usedResources=usedCapacity=0.0, numApps=0, numContainers=0
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initializeQueueMappings(436)) - Initialized queue
> mappings, override: false
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initScheduler(305)) - Initialized CapacityScheduler
> with calculator=class
> org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator,
> minimumAllocation=<>, maximumAllocation=< vCores:32>>, asynchronousScheduling=false, asyncScheduleInterval=5ms
> 2014-11-09 04:12:53,015 INFO service.AbstractService
> (AbstractService.java:noteFailure(272)) - Service ResourceManager failed in
> state STARTED; cause: java.lang.NullPointerException
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.addApplicationAttempt(CapacityScheduler.java:734)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1089)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AttemptRecoveredTransition.transition(RMAppAttemptImpl.java:1041)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AttemptRecoveredTransition.transition(RMAppAttemptImpl.java:1005)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:757)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:106)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.recoverAppAttempts(RMAppImpl.java:821)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.access$1900(RMAppImpl.java:101)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:843)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:826)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:701)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:312)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:413)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1207)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:590)
> at
> org.apache.hadoop.service.Abstra
[jira] [Commented] (YARN-2834) Resource manager crashed with Null Pointer Exception
[
https://issues.apache.org/jira/browse/YARN-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204128#comment-14204128
]
Vinod Kumar Vavilapalli commented on YARN-2834:
---
bq. Even in the regular case, RM doesn't fail the app if token renew fails, why
do we need to fail the app if token-renew fails on recovery.
After more discussions with [~jianhe] offline, for things like Timeline tokens
which are automatically obtained whether the app needs it or not (we should fix
this to be user driven), we can ignore failures. But for HDFS Tokens etc,
ignoring failures is bad because it (1) wastes resources as AMs will continue
and eventually fail (2) app doesn't know what happened it fails eventually.
Anyways, treating renewal failures is broken today. I am okay ignoring renewal
failures during recovery in this ticket. But let's file a blocker for handling
them correctly in 2.7.
> Resource manager crashed with Null Pointer Exception
>
>
> Key: YARN-2834
> URL: https://issues.apache.org/jira/browse/YARN-2834
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yesha Vora
>Assignee: Jian He
>Priority: Critical
> Attachments: YARN-2834.1.patch
>
>
> Resource manager failed after restart.
> {noformat}
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initializeQueues(467)) - Initialized root queue root:
> numChildQueue= 2, capacity=1.0, absoluteCapacity=1.0,
> usedResources=usedCapacity=0.0, numApps=0, numContainers=0
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initializeQueueMappings(436)) - Initialized queue
> mappings, override: false
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initScheduler(305)) - Initialized CapacityScheduler
> with calculator=class
> org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator,
> minimumAllocation=<>, maximumAllocation=< vCores:32>>, asynchronousScheduling=false, asyncScheduleInterval=5ms
> 2014-11-09 04:12:53,015 INFO service.AbstractService
> (AbstractService.java:noteFailure(272)) - Service ResourceManager failed in
> state STARTED; cause: java.lang.NullPointerException
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.addApplicationAttempt(CapacityScheduler.java:734)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1089)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AttemptRecoveredTransition.transition(RMAppAttemptImpl.java:1041)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AttemptRecoveredTransition.transition(RMAppAttemptImpl.java:1005)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:757)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:106)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.recoverAppAttempts(RMAppImpl.java:821)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.access$1900(RMAppImpl.java:101)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:843)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:826)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:701)
> at
[jira] [Updated] (YARN-342) RM doesn't retry token renewals
[
https://issues.apache.org/jira/browse/YARN-342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated YARN-342:
-
Priority: Critical (was: Major)
Target Version/s: 2.7.0 (was: 3.0.0, 2.1.0-beta)
This one's old but bad. Targeting fix in 2.7.
> RM doesn't retry token renewals
> ---
>
> Key: YARN-342
> URL: https://issues.apache.org/jira/browse/YARN-342
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 3.0.0, 2.0.0-alpha, 0.23.6
>Reporter: Daryn Sharp
>Priority: Critical
>
> The RM stops trying to renew tokens if any exception occurs during the renew.
> This should be changed to abort only if the exception is {{InvalidToken}} to
> allow resilience to transient network failures, issues associated with
> aborted connections when the NN is overloaded, cluster upgrades, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2834) Resource manager crashed with Null Pointer Exception
[
https://issues.apache.org/jira/browse/YARN-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated YARN-2834:
--
Priority: Critical (was: Major)
Target Version/s: 2.6.0
bq. The reason RM fails with NPE is that now if app fails to renew the token,
it'll move to FAILED state and save the app final state. But we didn't fail the
RMAppAttempt and save the attempt final state.
Shouldn't we move app-attempts to FAILED state whose parent app itself has
failed? That is consistent with what happens in the regular control path.
bq. Even in the regular case, RM doesn't fail the app if token renew fails, why
do we need to fail the app if token-renew fails on recovery.
Agreed this is an existing problem, but we should fix that too separately.
YARN-342 is related.
> Resource manager crashed with Null Pointer Exception
>
>
> Key: YARN-2834
> URL: https://issues.apache.org/jira/browse/YARN-2834
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yesha Vora
>Assignee: Jian He
>Priority: Critical
> Attachments: YARN-2834.1.patch
>
>
> Resource manager failed after restart.
> {noformat}
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initializeQueues(467)) - Initialized root queue root:
> numChildQueue= 2, capacity=1.0, absoluteCapacity=1.0,
> usedResources=usedCapacity=0.0, numApps=0, numContainers=0
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initializeQueueMappings(436)) - Initialized queue
> mappings, override: false
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initScheduler(305)) - Initialized CapacityScheduler
> with calculator=class
> org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator,
> minimumAllocation=<>, maximumAllocation=< vCores:32>>, asynchronousScheduling=false, asyncScheduleInterval=5ms
> 2014-11-09 04:12:53,015 INFO service.AbstractService
> (AbstractService.java:noteFailure(272)) - Service ResourceManager failed in
> state STARTED; cause: java.lang.NullPointerException
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.addApplicationAttempt(CapacityScheduler.java:734)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1089)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AttemptRecoveredTransition.transition(RMAppAttemptImpl.java:1041)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AttemptRecoveredTransition.transition(RMAppAttemptImpl.java:1005)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:757)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:106)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.recoverAppAttempts(RMAppImpl.java:821)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.access$1900(RMAppImpl.java:101)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:843)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:826)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:701)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:312)
[jira] [Updated] (YARN-2834) Resource manager crashed with Null Pointer Exception
[
https://issues.apache.org/jira/browse/YARN-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jian He updated YARN-2834:
--
Attachment: YARN-2834.1.patch
> Resource manager crashed with Null Pointer Exception
>
>
> Key: YARN-2834
> URL: https://issues.apache.org/jira/browse/YARN-2834
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yesha Vora
>Assignee: Jian He
> Attachments: YARN-2834.1.patch
>
>
> Resource manager failed after restart.
> {noformat}
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initializeQueues(467)) - Initialized root queue root:
> numChildQueue= 2, capacity=1.0, absoluteCapacity=1.0,
> usedResources=usedCapacity=0.0, numApps=0, numContainers=0
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initializeQueueMappings(436)) - Initialized queue
> mappings, override: false
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initScheduler(305)) - Initialized CapacityScheduler
> with calculator=class
> org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator,
> minimumAllocation=<>, maximumAllocation=< vCores:32>>, asynchronousScheduling=false, asyncScheduleInterval=5ms
> 2014-11-09 04:12:53,015 INFO service.AbstractService
> (AbstractService.java:noteFailure(272)) - Service ResourceManager failed in
> state STARTED; cause: java.lang.NullPointerException
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.addApplicationAttempt(CapacityScheduler.java:734)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1089)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AttemptRecoveredTransition.transition(RMAppAttemptImpl.java:1041)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AttemptRecoveredTransition.transition(RMAppAttemptImpl.java:1005)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:757)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:106)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.recoverAppAttempts(RMAppImpl.java:821)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.access$1900(RMAppImpl.java:101)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:843)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:826)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:701)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:312)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:413)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1207)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:590)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:1014)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1051)
> at
> org.ap
[jira] [Commented] (YARN-2834) Resource manager crashed with Null Pointer Exception
[
https://issues.apache.org/jira/browse/YARN-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204104#comment-14204104
]
Jian He commented on YARN-2834:
---
Talked with Zhijie, it should be fine to just ignore the token-renew exception
on recovery and let app continue. App itself could decide whether it choose to
fail or not when the token eventually expires. Even in the regular case, RM
doesn't fail the app if token renew fails, why do we need to fail the app if
token-renew fails on recovery.
> Resource manager crashed with Null Pointer Exception
>
>
> Key: YARN-2834
> URL: https://issues.apache.org/jira/browse/YARN-2834
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yesha Vora
>Assignee: Jian He
>
> Resource manager failed after restart.
> {noformat}
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initializeQueues(467)) - Initialized root queue root:
> numChildQueue= 2, capacity=1.0, absoluteCapacity=1.0,
> usedResources=usedCapacity=0.0, numApps=0, numContainers=0
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initializeQueueMappings(436)) - Initialized queue
> mappings, override: false
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initScheduler(305)) - Initialized CapacityScheduler
> with calculator=class
> org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator,
> minimumAllocation=<>, maximumAllocation=< vCores:32>>, asynchronousScheduling=false, asyncScheduleInterval=5ms
> 2014-11-09 04:12:53,015 INFO service.AbstractService
> (AbstractService.java:noteFailure(272)) - Service ResourceManager failed in
> state STARTED; cause: java.lang.NullPointerException
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.addApplicationAttempt(CapacityScheduler.java:734)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1089)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AttemptRecoveredTransition.transition(RMAppAttemptImpl.java:1041)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AttemptRecoveredTransition.transition(RMAppAttemptImpl.java:1005)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:757)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:106)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.recoverAppAttempts(RMAppImpl.java:821)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.access$1900(RMAppImpl.java:101)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:843)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:826)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:701)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:312)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:413)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1207)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:590)
> at
> org
[jira] [Commented] (YARN-2834) Resource manager crashed with Null Pointer Exception
[
https://issues.apache.org/jira/browse/YARN-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204101#comment-14204101
]
Jian He commented on YARN-2834:
---
The reason RM fails with NPE is that now if app fails to renew the token, it'll
move to FAILED state and save the app final state. But we didn't fail the
RMAppAttempt and save the attempt final state. So, on recovery, app won't be
added to scheduler because app is already in final state, but it'll still try
to add the attempt to scheduler which causes the NPE.
> Resource manager crashed with Null Pointer Exception
>
>
> Key: YARN-2834
> URL: https://issues.apache.org/jira/browse/YARN-2834
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yesha Vora
>Assignee: Jian He
>
> Resource manager failed after restart.
> {noformat}
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initializeQueues(467)) - Initialized root queue root:
> numChildQueue= 2, capacity=1.0, absoluteCapacity=1.0,
> usedResources=usedCapacity=0.0, numApps=0, numContainers=0
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initializeQueueMappings(436)) - Initialized queue
> mappings, override: false
> 2014-11-09 04:12:53,013 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:initScheduler(305)) - Initialized CapacityScheduler
> with calculator=class
> org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator,
> minimumAllocation=<>, maximumAllocation=< vCores:32>>, asynchronousScheduling=false, asyncScheduleInterval=5ms
> 2014-11-09 04:12:53,015 INFO service.AbstractService
> (AbstractService.java:noteFailure(272)) - Service ResourceManager failed in
> state STARTED; cause: java.lang.NullPointerException
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.addApplicationAttempt(CapacityScheduler.java:734)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1089)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AttemptRecoveredTransition.transition(RMAppAttemptImpl.java:1041)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AttemptRecoveredTransition.transition(RMAppAttemptImpl.java:1005)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:757)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:106)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.recoverAppAttempts(RMAppImpl.java:821)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.access$1900(RMAppImpl.java:101)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:843)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:826)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:701)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:312)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:413)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1207)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:590)
[jira] [Commented] (YARN-2830) Add backwords compatible ContainerId.newInstance constructor for use within Tez Local Mode
[ https://issues.apache.org/jira/browse/YARN-2830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204074#comment-14204074 ] Tsuyoshi OZAWA commented on YARN-2830: -- +1, the failure of TestApplicationClientProtocolOnHA is not related to the patch and succeeded on my local. Vinod, thanks for pointing the link. > Add backwords compatible ContainerId.newInstance constructor for use within > Tez Local Mode > -- > > Key: YARN-2830 > URL: https://issues.apache.org/jira/browse/YARN-2830 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Blocker > Attachments: YARN-2830-v1.patch, YARN-2830-v2.patch, > YARN-2830-v3.patch, YARN-2830-v4.patch > > > YARN-2229 modified the private unstable api for constructing. Tez uses this > api (shouldn't, but does) for use with Tez Local Mode. This causes a > NoSuchMethod error when using Tez compiled against pre-2.6. Instead I propose > we add the backwards compatible api since overflow is not a problem in tez > local mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2607) TestDistributedShell fails in trunk
[
https://issues.apache.org/jira/browse/YARN-2607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204073#comment-14204073
]
Tsuyoshi OZAWA commented on YARN-2607:
--
TestDistributedShellWithNodeLabels.testDSShellWithNodeLabelExpression still
fails on my local because of timeout.
> TestDistributedShell fails in trunk
> ---
>
> Key: YARN-2607
> URL: https://issues.apache.org/jira/browse/YARN-2607
> Project: Hadoop YARN
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Wangda Tan
> Fix For: 2.6.0
>
> Attachments: YARN-2607-1.patch, YARN-2607-2.patch, YARN-2607-3.patch
>
>
> From https://builds.apache.org/job/Hadoop-Yarn-trunk/691/console :
> {code}
> testDSRestartWithPreviousRunningContainers(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 35.641 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSRestartWithPreviousRunningContainers(TestDistributedShell.java:308)
> {code}
> On Linux, I got the following locally:
> {code}
> testDSAttemptFailuresValidityIntervalFailed(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 64.715 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSAttemptFailuresValidityIntervalFailed(TestDistributedShell.java:384)
> testDSAttemptFailuresValidityIntervalSucess(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 115.842 sec <<< ERROR!
> java.lang.Exception: test timed out after 9 milliseconds
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.yarn.applications.distributedshell.Client.monitorApplication(Client.java:680)
> at
> org.apache.hadoop.yarn.applications.distributedshell.Client.run(Client.java:661)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSAttemptFailuresValidityIntervalSucess(TestDistributedShell.java:342)
> testDSRestartWithPreviousRunningContainers(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 35.633 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSRestartWithPreviousRunningContainers(TestDistributedShell.java:308)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2835) YARN WebApps should be a public API
[ https://issues.apache.org/jira/browse/YARN-2835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated YARN-2835: -- Target Version/s: 2.6.0 (was: 2.7.0) > YARN WebApps should be a public API > --- > > Key: YARN-2835 > URL: https://issues.apache.org/jira/browse/YARN-2835 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Hitesh Shah > > For application masters that need to host webservices and/or a UI, this is > common functionality that could be re-used across the ecosystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2835) YARN WebApps should be a public API
[ https://issues.apache.org/jira/browse/YARN-2835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated YARN-2835: -- Target Version/s: 2.7.0 > YARN WebApps should be a public API > --- > > Key: YARN-2835 > URL: https://issues.apache.org/jira/browse/YARN-2835 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Hitesh Shah > > For application masters that need to host webservices and/or a UI, this is > common functionality that could be re-used across the ecosystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2835) YARN WebApps should be a public API
Hitesh Shah created YARN-2835: - Summary: YARN WebApps should be a public API Key: YARN-2835 URL: https://issues.apache.org/jira/browse/YARN-2835 Project: Hadoop YARN Issue Type: Bug Reporter: Hitesh Shah For application masters that need to host webservices and/or a UI, this is common functionality that could be re-used across the ecosystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2830) Add backwords compatible ContainerId.newInstance constructor for use within Tez Local Mode
[ https://issues.apache.org/jira/browse/YARN-2830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204006#comment-14204006 ] Tsuyoshi OZAWA commented on YARN-2830: -- [~acmurthy] Let me review the latest patch. > Add backwords compatible ContainerId.newInstance constructor for use within > Tez Local Mode > -- > > Key: YARN-2830 > URL: https://issues.apache.org/jira/browse/YARN-2830 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Blocker > Attachments: YARN-2830-v1.patch, YARN-2830-v2.patch, > YARN-2830-v3.patch, YARN-2830-v4.patch > > > YARN-2229 modified the private unstable api for constructing. Tez uses this > api (shouldn't, but does) for use with Tez Local Mode. This causes a > NoSuchMethod error when using Tez compiled against pre-2.6. Instead I propose > we add the backwards compatible api since overflow is not a problem in tez > local mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2830) Add backwords compatible ContainerId.newInstance constructor for use within Tez Local Mode
[ https://issues.apache.org/jira/browse/YARN-2830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204002#comment-14204002 ] Arun C Murthy commented on YARN-2830: - Is this ready to go? > Add backwords compatible ContainerId.newInstance constructor for use within > Tez Local Mode > -- > > Key: YARN-2830 > URL: https://issues.apache.org/jira/browse/YARN-2830 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Blocker > Attachments: YARN-2830-v1.patch, YARN-2830-v2.patch, > YARN-2830-v3.patch, YARN-2830-v4.patch > > > YARN-2229 modified the private unstable api for constructing. Tez uses this > api (shouldn't, but does) for use with Tez Local Mode. This causes a > NoSuchMethod error when using Tez compiled against pre-2.6. Instead I propose > we add the backwards compatible api since overflow is not a problem in tez > local mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2607) TestDistributedShell fails in trunk
[
https://issues.apache.org/jira/browse/YARN-2607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14203956#comment-14203956
]
Hudson commented on YARN-2607:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1952 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1952/])
YARN-2607. Fixed issues in TestDistributedShell. Contributed by Wangda Tan.
(vinodkv: rev 737d9284c109dac20ff423f30c62f6abe2db10f7)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShellWithNodeLabels.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java
* hadoop-yarn-project/CHANGES.txt
> TestDistributedShell fails in trunk
> ---
>
> Key: YARN-2607
> URL: https://issues.apache.org/jira/browse/YARN-2607
> Project: Hadoop YARN
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Wangda Tan
> Fix For: 2.6.0
>
> Attachments: YARN-2607-1.patch, YARN-2607-2.patch, YARN-2607-3.patch
>
>
> From https://builds.apache.org/job/Hadoop-Yarn-trunk/691/console :
> {code}
> testDSRestartWithPreviousRunningContainers(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 35.641 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSRestartWithPreviousRunningContainers(TestDistributedShell.java:308)
> {code}
> On Linux, I got the following locally:
> {code}
> testDSAttemptFailuresValidityIntervalFailed(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 64.715 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSAttemptFailuresValidityIntervalFailed(TestDistributedShell.java:384)
> testDSAttemptFailuresValidityIntervalSucess(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 115.842 sec <<< ERROR!
> java.lang.Exception: test timed out after 9 milliseconds
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.yarn.applications.distributedshell.Client.monitorApplication(Client.java:680)
> at
> org.apache.hadoop.yarn.applications.distributedshell.Client.run(Client.java:661)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSAttemptFailuresValidityIntervalSucess(TestDistributedShell.java:342)
> testDSRestartWithPreviousRunningContainers(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 35.633 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSRestartWithPreviousRunningContainers(TestDistributedShell.java:308)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2607) TestDistributedShell fails in trunk
[
https://issues.apache.org/jira/browse/YARN-2607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14203928#comment-14203928
]
Hudson commented on YARN-2607:
--
SUCCESS: Integrated in Hadoop-Hdfs-trunk #1928 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1928/])
YARN-2607. Fixed issues in TestDistributedShell. Contributed by Wangda Tan.
(vinodkv: rev 737d9284c109dac20ff423f30c62f6abe2db10f7)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShellWithNodeLabels.java
* hadoop-yarn-project/CHANGES.txt
> TestDistributedShell fails in trunk
> ---
>
> Key: YARN-2607
> URL: https://issues.apache.org/jira/browse/YARN-2607
> Project: Hadoop YARN
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Wangda Tan
> Fix For: 2.6.0
>
> Attachments: YARN-2607-1.patch, YARN-2607-2.patch, YARN-2607-3.patch
>
>
> From https://builds.apache.org/job/Hadoop-Yarn-trunk/691/console :
> {code}
> testDSRestartWithPreviousRunningContainers(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 35.641 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSRestartWithPreviousRunningContainers(TestDistributedShell.java:308)
> {code}
> On Linux, I got the following locally:
> {code}
> testDSAttemptFailuresValidityIntervalFailed(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 64.715 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSAttemptFailuresValidityIntervalFailed(TestDistributedShell.java:384)
> testDSAttemptFailuresValidityIntervalSucess(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 115.842 sec <<< ERROR!
> java.lang.Exception: test timed out after 9 milliseconds
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.yarn.applications.distributedshell.Client.monitorApplication(Client.java:680)
> at
> org.apache.hadoop.yarn.applications.distributedshell.Client.run(Client.java:661)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSAttemptFailuresValidityIntervalSucess(TestDistributedShell.java:342)
> testDSRestartWithPreviousRunningContainers(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 35.633 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSRestartWithPreviousRunningContainers(TestDistributedShell.java:308)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2607) TestDistributedShell fails in trunk
[
https://issues.apache.org/jira/browse/YARN-2607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14203885#comment-14203885
]
Hudson commented on YARN-2607:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #738 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/738/])
YARN-2607. Fixed issues in TestDistributedShell. Contributed by Wangda Tan.
(vinodkv: rev 737d9284c109dac20ff423f30c62f6abe2db10f7)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShellWithNodeLabels.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java
> TestDistributedShell fails in trunk
> ---
>
> Key: YARN-2607
> URL: https://issues.apache.org/jira/browse/YARN-2607
> Project: Hadoop YARN
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Wangda Tan
> Fix For: 2.6.0
>
> Attachments: YARN-2607-1.patch, YARN-2607-2.patch, YARN-2607-3.patch
>
>
> From https://builds.apache.org/job/Hadoop-Yarn-trunk/691/console :
> {code}
> testDSRestartWithPreviousRunningContainers(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 35.641 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSRestartWithPreviousRunningContainers(TestDistributedShell.java:308)
> {code}
> On Linux, I got the following locally:
> {code}
> testDSAttemptFailuresValidityIntervalFailed(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 64.715 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSAttemptFailuresValidityIntervalFailed(TestDistributedShell.java:384)
> testDSAttemptFailuresValidityIntervalSucess(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 115.842 sec <<< ERROR!
> java.lang.Exception: test timed out after 9 milliseconds
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.yarn.applications.distributedshell.Client.monitorApplication(Client.java:680)
> at
> org.apache.hadoop.yarn.applications.distributedshell.Client.run(Client.java:661)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSAttemptFailuresValidityIntervalSucess(TestDistributedShell.java:342)
> testDSRestartWithPreviousRunningContainers(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 35.633 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSRestartWithPreviousRunningContainers(TestDistributedShell.java:308)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
