Bibin A Chundatt created YARN-3754:
--------------------------------------
Summary: Race condition when the NodeManager is shutting down and
container is launched
Key: YARN-3754
URL: https://issues.apache.org/jira/browse/YARN-3754
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Environment: Suse 11 Sp3
Reporter: Bibin A Chundatt
Container is launched and returned to ContainerImpl
NodeManager closed the DB connection which resulting in
{{org.iq80.leveldb.DBException: Closed}}.
*Attaching the exception trace*
{code}
2015-05-30 02:11:49,122 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
Unable to update state store diagnostics for
container_e310_1432817693365_3338_01_000002
java.io.IOException: org.iq80.leveldb.DBException: Closed
at
org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.storeContainerDiagnostics(NMLeveldbStateStoreService.java:261)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl$ContainerDiagnosticsUpdateTransition.transition(ContainerImpl.java:1109)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl$ContainerDiagnosticsUpdateTransition.transition(ContainerImpl.java:1101)
at
org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362)
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:1129)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:83)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:246)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.iq80.leveldb.DBException: Closed
at org.fusesource.leveldbjni.internal.JniDB.put(JniDB.java:123)
at org.fusesource.leveldbjni.internal.JniDB.put(JniDB.java:106)
at
org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.storeContainerDiagnostics(NMLeveldbStateStoreService.java:259)
... 15 more
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
