[
https://issues.apache.org/jira/browse/YARN-845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13685988#comment-13685988
]
Arpit Gupta commented on YARN-845:
----------------------------------
Mayank
Unfortunately cannot consistently reproduce this. We saw this on a cluster
where pig e2e tests were running and started failing because the RM went down.
On other nights and other clusters the same set of tests have gone through
without any issues.
> RM crash with NPE on NODE_UPDATE
> --------------------------------
>
> Key: YARN-845
> URL: https://issues.apache.org/jira/browse/YARN-845
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
> Affects Versions: 3.0.0, 2.1.0-beta
> Reporter: Arpit Gupta
>
> the following stack trace is generated in rm
> {code}
> n, service: 68.142.246.147:45454 }, ] resource=<memory:1536, vCores:1>
> queue=default: capacity=1.0, absoluteCapacity=1.0,
> usedResources=<memory:44544, vCores:29>usedCapacity=0.90625,
> absoluteUsedCapacity=0.90625, numApps=1, numContainers=29
> usedCapacity=0.90625 absoluteUsedCapacity=0.90625 used=<memory:44544,
> vCores:29> cluster=<memory:49152, vCores:48>
> 2013-06-17 12:43:53,655 INFO capacity.ParentQueue
> (ParentQueue.java:completedContainer(696)) - completedContainer queue=root
> usedCapacity=0.90625 absoluteUsedCapacity=0.90625 used=<memory:44544,
> vCores:29> cluster=<memory:49152, vCores:48>
> 2013-06-17 12:43:53,656 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:completedContainer(832)) - Application
> appattempt_1371448527090_0844_000001 released container
> container_1371448527090_0844_01_000005 on node: host:
> hor15n00.gq1.ygridcore.net:45454 #containers=4 available=2048 used=6144 with
> event: FINISHED
> 2013-06-17 12:43:53,656 INFO capacity.CapacityScheduler
> (CapacityScheduler.java:nodeUpdate(661)) - Trying to fulfill reservation for
> application application_1371448527090_0844 on node:
> hor15n00.gq1.ygridcore.net:45454
> 2013-06-17 12:43:53,656 INFO fica.FiCaSchedulerApp
> (FiCaSchedulerApp.java:unreserve(435)) - Application
> application_1371448527090_0844 unreserved on node host:
> hor15n00.gq1.ygridcore.net:45454 #containers=4 available=2048 used=6144,
> currently has 4 at priority 20; currentReservation <memory:6144, vCores:4>
> 2013-06-17 12:43:53,656 INFO scheduler.AppSchedulingInfo
> (AppSchedulingInfo.java:updateResourceRequests(168)) - checking for
> deactivate...
> 2013-06-17 12:43:53,657 FATAL resourcemanager.ResourceManager
> (ResourceManager.java:run(422)) - Error in handling event type NODE_UPDATE to
> the scheduler
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.unreserve(FiCaSchedulerApp.java:432)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.unreserve(LeafQueue.java:1416)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1346)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1221)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1180)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignReservedContainer(LeafQueue.java:939)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:803)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:665)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:727)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:83)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:413)
> at java.lang.Thread.run(Thread.java:662)
> 2013-06-17 12:43:53,659 INFO resourcemanager.ResourceManager
> (ResourceManager.java:run(426)) - Exiting, bbye..
> 2013-06-17 12:43:53,665 INFO mortbay.log (Slf4jLog.java:info(67)) - Stopped
> selectchannelconnec...@hor14n33.gq1.ygridcore.net:8088
> 2013-06-17 12:43:53,765 ERROR delegation.AbstractDelegationTokenSecretManager
> (AbstractDelegationTokenSecretManager.java:run(513)) - InterruptedExcpetion
> recieved for ExpiredTokenRemover thread java.lang.InterruptedException: sleep
> interrupted
> 2013-06-17 12:43:53,766 INFO impl.MetricsSystemImpl
> (MetricsSystemImpl.java:stop(200)) - Stopping ResourceManager metrics
> system...
> 2013-06-17 12:43:53,767 INFO impl.MetricsSystemImpl
> (MetricsSystemImpl.java:stop(206)) - ResourceManager metrics system stopped.
> 2013-06-17 12:43:53,767 INFO impl.MetricsSystemImpl
> (MetricsSystemImpl.java:shutdown(572)) - ResourceManager metrics system
> shutdown complete.
> 2013-06-17 12:43:53,768 WARN amlauncher.ApplicationMasterLauncher
> (ApplicationMasterLauncher.java:run(98)) -
> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
> interrupted. Returning.
> 2013-06-17 12:43:53,768 INFO ipc.Server (Server.java:stop(2167)) - Stopping
> server on 8033
> 2013-06-17 12:43:53,770 INFO ipc.Server (Server.java:run(686)) - Stopping
> IPC Server listener on 8033
> 2013-06-17 12:43:53,770 INFO ipc.Server (Server.java:stop(2167)) - Stopping
> server on 8032
> 2013-06-17 12:43:53,770 INFO ipc.Server (Server.java:run(828)) - Stopping
> IPC Server Responder
> 2013-06-17 12:43:53,771 INFO ipc.Server (Server.java:run(686)) - Stopping
> IPC Server listener on 8032
> 2013-06-17 12:43:53,771 INFO ipc.Server (Server.java:run(828)) - Stopping
> IPC Server Responder
> 2013-06-17 12:43:53,771 INFO ipc.Server (Server.java:stop(2167)) - Stopping
> server on 8030
> 2013-06-17 12:43:53,773 INFO ipc.Server (Server.java:run(686)) - Stopping
> IPC Server listener on 8030
> 2013-06-17 12:43:53,773 INFO ipc.Server (Server.java:stop(2167)) - Stopping
> server on 8031
> 2013-06-17 12:43:53,773 INFO ipc.Server (Server.java:run(828)) - Stopping
> IPC Server Responder
> 2013-06-17 12:43:53,774 INFO ipc.Server (Server.java:run(686)) - Stopping
> IPC Server listener on 8031
> 2013-06-17 12:43:53,775 INFO ipc.Server (Server.java:run(828)) - Stopping
> IPC Server Responder
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
