YarnException: Failed to check for existence of remoteLogDir [/var/log/hadoop-yarn/apps]
I'm not very familiar with Yarn, but are you sure that directory exists? Also https://issues.apache.org/jira/browse/YARN-24 states that the namenode might be unreachable There was also this link https://groups.google.com/a/cloudera.org/forum/#!topic/cdh-user/UOYbUcK_6q4 which will probably be more useful. Sorry if this doesn't help. On Fri, Jul 25, 2014 at 11:30 AM, David Fryer <[email protected]> wrote: > Hi Bigtop, > > I have used puppet to deploy Hadoop onto a master and three slaves, > however on the slaves, nodemanager will not start. > > The errors found in /var/log/hadoop-yarn/yarn-yarn-nodemanager-slave1.log > are here: > > 2014-07-26 01:24:22,370 ERROR >> org.apache.hadoop.yarn.service.CompositeService: Error starting services >> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl >> org.apache.hadoop.yarn.YarnException: Failed to check for existence of >> remoteLogDir [/var/log/hadoop-yarn/apps] >> at >> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.verifyAndCreateRemoteLogDir(LogAggregationService.java:179) >> at >> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.start(LogAggregationService.java:132) >> at >> org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) >> at >> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.start(ContainerManagerImpl.java:248) >> at >> org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) >> at >> org.apache.hadoop.yarn.server.nodemanager.NodeManager.start(NodeManager.java:199) >> at >> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:322) >> at >> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:359) >> 2014-07-26 01:24:22,372 INFO >> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: >> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService >> waiting for pending aggregation during exit >> 2014-07-26 01:24:22,372 INFO >> org.apache.hadoop.yarn.service.AbstractService: Service:Dispatcher is >> stopped. >> 2014-07-26 01:24:22,372 WARN >> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: >> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl >> is interrupted. Exiting. >> 2014-07-26 01:24:22,372 INFO >> org.apache.hadoop.yarn.service.AbstractService: Service:containers-monitor >> is stopped. >> 2014-07-26 01:24:22,379 INFO >> org.apache.hadoop.yarn.service.AbstractService: Service:httpshuffle is >> stopped. >> 2014-07-26 01:24:22,379 INFO >> org.apache.hadoop.yarn.service.AbstractService: >> Service:org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices >> is stopped. >> 2014-07-26 01:24:22,379 INFO >> org.apache.hadoop.yarn.service.AbstractService: Service:containers-launcher >> is stopped. >> 2014-07-26 01:24:22,379 INFO org.apache.hadoop.ipc.Server: Stopping >> server on 8040 >> 2014-07-26 01:24:22,380 INFO org.apache.hadoop.ipc.Server: Stopping IPC >> Server listener on 8040 >> 2014-07-26 01:24:22,380 INFO org.apache.hadoop.ipc.Server: Stopping IPC >> Server Responder >> 2014-07-26 01:24:22,380 INFO >> org.apache.hadoop.yarn.service.AbstractService: >> Service:org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker >> is stopped. >> 2014-07-26 01:24:22,380 INFO >> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: >> Public cache exiting >> 2014-07-26 01:24:22,380 INFO >> org.apache.hadoop.yarn.service.AbstractService: >> Service:org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService >> is stopped. >> 2014-07-26 01:24:22,380 ERROR >> org.apache.hadoop.yarn.service.CompositeService: Error starting services >> org.apache.hadoop.yarn.server.nodemanager.NodeManager >> org.apache.hadoop.yarn.YarnException: Failed to Start >> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl >> at >> org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:78) >> at >> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.start(ContainerManagerImpl.java:248) >> at >> org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) >> at >> org.apache.hadoop.yarn.server.nodemanager.NodeManager.start(NodeManager.java:199) >> at >> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:322) >> at >> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:359) >> Caused by: org.apache.hadoop.yarn.YarnException: Failed to check for >> existence of remoteLogDir [/var/log/hadoop-yarn/apps] >> at >> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.verifyAndCreateRemoteLogDir(LogAggregationService.java:179) >> at >> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.start(LogAggregationService.java:132) >> at >> org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) >> ... 5 more >> 2014-07-26 01:24:22,380 INFO org.apache.hadoop.ipc.Server: Stopping >> server on 42208 >> 2014-07-26 01:24:22,380 INFO org.apache.hadoop.ipc.Server: Stopping IPC >> Server listener on 42208 >> 2014-07-26 01:24:22,381 INFO org.apache.hadoop.ipc.Server: Stopping IPC >> Server Responder >> 2014-07-26 01:24:22,380 INFO >> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: >> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService >> waiting for pending aggregation during exit >> 2014-07-26 01:24:22,381 INFO org.apache.hadoop.ipc.Server: Stopping >> server on 8040 >> 2014-07-26 01:24:22,381 INFO >> org.apache.hadoop.yarn.service.AbstractService: >> Service:org.apache.hadoop.yarn.server.nodemanager.NodeResourceMonitorImpl >> is stopped. >> 2014-07-26 01:24:22,381 INFO >> org.apache.hadoop.yarn.service.AbstractService: >> Service:org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService >> is stopped. >> 2014-07-26 01:24:22,381 INFO >> org.apache.hadoop.yarn.service.AbstractService: >> Service:org.apache.hadoop.yarn.server.nodemanager.NodeHealthCheckerService >> is stopped. >> 2014-07-26 01:24:22,381 INFO >> org.apache.hadoop.yarn.service.AbstractService: >> Service:org.apache.hadoop.yarn.server.nodemanager.DeletionService is >> stopped. >> 2014-07-26 01:24:22,381 FATAL >> org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting >> NodeManager >> org.apache.hadoop.yarn.YarnException: Failed to Start >> org.apache.hadoop.yarn.server.nodemanager.NodeManager >> at >> org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:78) >> at >> org.apache.hadoop.yarn.server.nodemanager.NodeManager.start(NodeManager.java:199) >> at >> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:322) >> at >> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:359) >> Caused by: org.apache.hadoop.yarn.YarnException: Failed to Start >> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl >> at >> org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:78) >> at >> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.start(ContainerManagerImpl.java:248) >> at >> org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) >> ... 3 more >> Caused by: org.apache.hadoop.yarn.YarnException: Failed to check for >> existence of remoteLogDir [/var/log/hadoop-yarn/apps] >> at >> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.verifyAndCreateRemoteLogDir(LogAggregationService.java:179) >> at >> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.start(LogAggregationService.java:132) >> at >> org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) >> ... 5 more >> 2014-07-26 01:24:22,382 INFO org.apache.hadoop.ipc.Server: Stopping >> server on 42208 >> 2014-07-26 01:24:22,382 INFO >> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: >> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService >> waiting for pending aggregation during exit >> 2014-07-26 01:24:22,382 INFO org.apache.hadoop.ipc.Server: Stopping >> server on 8040 >> 2014-07-26 01:24:22,382 INFO >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NodeManager >> metrics system... >> 2014-07-26 01:24:22,383 INFO >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics >> system stopped. >> 2014-07-26 01:24:22,383 INFO >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics >> system shutdown complete. >> 2014-07-26 01:24:22,385 INFO >> org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG: >> /************************************************************ >> SHUTDOWN_MSG: Shutting down NodeManager at slave1/127.0.0.1 >> ************************************************************/ > > > What can I do to fix this? > > -David Fryer > >
