[
https://issues.apache.org/jira/browse/YARN-5195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15312989#comment-15312989
]
Wangda Tan commented on YARN-5195:
----------------------------------
Investigated this issue, this only happens when async scheduling enabled,
container allocated to a node after the node removed from scheduler:
Logs look like:
{code}
2016-05-28 15:45:18,502 [ResourceManager Event Processor] INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Released
container container_1464449118385_0006_01_000324 of capacity <memory:2048,
vCores:1> on host cn042-10.l42scl.hortonworks.com:49161, which currently has 0
containers, <memory:0, vCores:0> used and <memory:49152, vCores:12> available,
release resources=true
2016-05-28 15:45:18,503 [ResourceManager Event Processor] INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Removed node node-1:49161 clusterResource: <memory:442368, vCores:108>
2016-05-28 15:45:18,526 [Thread-12] INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Assigned
container container_1464449118385_0006_01_000382 of capacity <memory:2048,
vCores:1> on host node-1:49161, which has 1 containers, <memory:2048, vCores:1>
used and <memory:47104, vCores:11> available after allocation
{code}
Add additional lock protection to async scheduling thread could prevent this
happen.
> RM crashed with NPE while handling APP_ATTEMPT_REMOVED event
> ------------------------------------------------------------
>
> Key: YARN-5195
> URL: https://issues.apache.org/jira/browse/YARN-5195
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
> Reporter: Karam Singh
> Assignee: Wangda Tan
>
> While running gridmix experiments one time came across incident where RM went
> down with following exception
> {noformat}
> 2016-05-28 15:45:24,459 [ResourceManager Event Processor] FATAL
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in
> handling event type APP_ATTEMPT_REMOVED to the scheduler
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.completedContainer(LeafQueue.java:1282)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.completedContainerInternal(CapacityScheduler.java:1469)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.completedContainer(AbstractYarnScheduler.java:497)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.doneApplicationAttempt(CapacityScheduler.java:860)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1319)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:127)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:704)
> at java.lang.Thread.run(Thread.java:745)
> 2016-05-28 15:45:24,460 [ApplicationMasterLauncher #49] INFO
> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Cleaning
> master appattempt_1464449118385_0006_000001
> 2016-05-28 15:45:24,460 [ResourceManager Event Processor] INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, bbye..
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
