yeah... there's not much there: -bash-4.1$ cd nm-local-dir/ -bash-4.1$ ll * filecache: total 0
nmPrivate: total 0 usercache: total 0 I'm using Open JDK, would that be a problem? More log: STARTUP_MSG: java = 1.7.0_75 ************************************************************/ 2015-04-19 14:38:58,168 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: registered UNIX signal handlers for [TERM, HUP, INT] 2015-04-19 14:38:58,562 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2015-04-19 14:38:59,018 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher 2015-04-19 14:38:59,020 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher 2015-04-19 14:38:59,021 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizationEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService 2015-04-19 14:38:59,021 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServicesEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices 2015-04-19 14:38:59,022 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl 2015-04-19 14:38:59,023 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher 2015-04-19 14:38:59,054 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.ContainerManagerEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl 2015-04-19 14:38:59,054 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.NodeManagerEventType for class org.apache.hadoop.yarn.server.nodemanager.NodeManager 2015-04-19 14:38:59,109 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 2015-04-19 14:38:59,197 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2015-04-19 14:38:59,197 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system started 2015-04-19 14:38:59,217 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.event.LogHandlerEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler 2015-04-19 14:38:59,217 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: per directory file limit = 8192 2015-04-19 14:38:59,227 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizerEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker 2015-04-19 14:38:59,248 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The Auxilurary Service named 'mapreduce_shuffle' in the configuration is for class class org.apache.hadoop.mapred.ShuffleHandler which has a name of 'httpshuffle'. Because these are not the same tools trying to send ServiceData and read Service Meta Data may have issues unless the refer to the name in the config. 2015-04-19 14:38:59,248 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Adding auxiliary service httpshuffle, "mapreduce_shuffle" 2015-04-19 14:38:59,281 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@7fc514a7 2015-04-19 14:38:59,281 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Using ResourceCalculatorProcessTree : null 2015-04-19 14:38:59,281 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Physical memory check enabled: true 2015-04-19 14:38:59,281 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Virtual memory check enabled: true 2015-04-19 14:38:59,284 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: NodeManager configured with 14 G physical memory allocated to containers, which is more than 80% of the total physical memory available (14.7 G). Thrashing might happen. 2015-04-19 14:38:59,287 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Initialized nodemanager for null: physical-memory=14336 virtual-memory=30106 virtual-cores=8 2015-04-19 14:38:59,318 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue 2015-04-19 14:38:59,334 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 38230 2015-04-19 14:38:59,359 INFO org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.yarn.api.ContainerManagementProtocolPB to the server 2015-04-19 14:38:59,359 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Blocking new container-requests as container manager rpc server is still starting. 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 38230: starting 2015-04-19 14:38:59,366 INFO org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager: Updating node address : ip-10-100-70-199.ec2.internal:38230 2015-04-19 14:38:59,372 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue 2015-04-19 14:38:59,373 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 8040 2015-04-19 14:38:59,376 INFO org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.yarn.server.nodemanager.api.LocalizationProtocolPB to the server 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8040: starting 2015-04-19 14:38:59,380 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer started on port 8040 2015-04-19 14:38:59,391 INFO org.apache.hadoop.mapred.IndexCache: IndexCache created with max memory = 10485760 2015-04-19 14:38:59,403 INFO org.apache.hadoop.mapred.ShuffleHandler: httpshuffle listening on port 13562 2015-04-19 14:38:59,405 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: ContainerManager started at datanode-03.prod.com/10.100.70.199:38230 2015-04-19 14:38:59,405 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: ContainerManager bound to 0.0.0.0/0.0.0.0:0 2015-04-19 14:38:59,405 INFO org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating NMWebApp at 0.0.0.0:8042 2015-04-19 14:38:59,471 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2015-04-19 14:38:59,475 INFO org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.nodemanager is not defined 2015-04-19 14:38:59,487 INFO org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context node 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding path spec: /node/* 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding path spec: /ws/* 2015-04-19 14:38:59,505 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 8042 2015-04-19 14:38:59,505 INFO org.mortbay.log: jetty-6.1.26 2015-04-19 14:38:59,545 INFO org.mortbay.log: Extract jar:file:/opt/test/service/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.6.0.jar!/webapps/node to /tmp/Jetty_0_0_0_0_8042_node____19tj0x/webapp 2015-04-19 14:38:59,778 INFO org.mortbay.log: Started HttpServer2$ [email protected]:8042 2015-04-19 14:38:59,778 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app /node started at 8042 2015-04-19 14:39:00,093 INFO org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules 2015-04-19 14:39:00,126 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending out 0 NM container statuses: [] 2015-04-19 14:39:00,131 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registering with RM using containers :[] 2015-04-19 14:39:00,176 INFO org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager: Rolling master-key for container-tokens, got key with id -543066608 2015-04-19 14:39:00,178 INFO org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM: Rolling master-key for container-tokens, got key with id -1243797706 2015-04-19 14:39:00,179 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered with ResourceManager as ip-10-100-70-199.ec2.internal:38230 with total resource of <memory:14336, vCores:8> 2015-04-19 14:39:00,179 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying ContainerManager to unblock new container-requests 2015-04-19 19:22:17,729 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1429450734039_0010_000001 (auth:SIMPLE) 2015-04-19 19:22:17,807 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_1429450734039_0010_01_000001 by user nobody 2015-04-19 19:22:17,828 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1429450734039_0010 2015-04-19 19:22:17,834 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody IP=10.100.66.251 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1429450734039_0010 CONTAINERID=container_1429450734039_0010_01_000001 2015-04-19 19:22:17,835 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1429450734039_0010 transitioned from NEW to INITING 2015-04-19 19:22:17,835 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Adding container_1429450734039_0010_01_000001 to application application_1429450734039_0010 2015-04-19 19:22:17,839 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1429450734039_0010 transitioned from INITING to RUNNING 2015-04-19 19:22:17,843 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1429450734039_0010_01_000001 transitioned from NEW to LOCALIZING 2015-04-19 19:22:17,843 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1429450734039_0010 2015-04-19 19:22:17,876 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.jar transitioned from INIT to DOWNLOADING 2015-04-19 19:22:17,877 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.splitmetainfo transitioned from INIT to DOWNLOADING 2015-04-19 19:22:17,877 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.split transitioned from INIT to DOWNLOADING 2015-04-19 19:22:17,877 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.xml transitioned from INIT to DOWNLOADING 2015-04-19 19:22:17,877 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_1429450734039_0010_01_000001 2015-04-19 19:22:17,880 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed java.lang.NullPointerException at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075) 2015-04-19 19:22:17,882 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1429450734039_0010_01_000001 transitioned from LOCALIZING to LOCALIZATION_FAILED 2015-04-19 19:22:17,886 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1429450734039_0010 CONTAINERID=container_1429450734039_0010_01_000001 2015-04-19 19:22:17,889 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1429450734039_0010_01_000001 transitioned from LOCALIZATION_FAILED to DONE 2015-04-19 19:22:17,889 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Removing container_1429450734039_0010_01_000001 from application application_1429450734039_0010 On Sun, Apr 19, 2015 at 1:16 PM, Brahma Reddy Battula < [email protected]> wrote: > As Alexander Alten-Lorenz pointed, it mostly config > issue(yarn.nodemanager.local-dirs > or mapred.local.dir).. > > can you able provide full logs..? > > Bytheway NPE is handled in Trunk ..Please check HADOOP-8436 for more > details.. > > ------------------------------ > From: [email protected] > Subject: Re: ResourceLocalizationService: Localizer failed when running pi > example > Date: Sun, 19 Apr 2015 17:59:13 +0200 > To: [email protected] > > > As you said, that looks like a config issue. I would spot on the NM's > local scratch dir (yarn.nodemanager.local-dirs). > > But without a complete stack trace, its a blind call. > > BR, > AL > > -- > mapredit.blogspot.com > > On Apr 18, 2015, at 6:24 PM, Fernando O. <[email protected]> wrote: > > Hey All, > It's me again with another noob question: I deployed a cluster (HA > mode) everything looked good but when I tried to run the pi example: > > bin/hadoop jar > ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100 > > the same error occurs if I try to generate data with teragen 100000000 > /test/data > > > 2015-04-18 15:49:04,090 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: > Localizer failed > java.lang.NullPointerException > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268) > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115) > at > org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075) > > > I'm guessing it's a configuration issue but I don't know what am I missing > :S > > >
