Hi, guess the "yarn.nodemanager.local-dirs" property is the problem. Can you provide that part of yarn-site.xml?
Thanks. Drake 민영근 Ph.D kt NexR On Mon, Apr 20, 2015 at 4:27 AM, Fernando O. <[email protected]> wrote: > yeah... there's not much there: > > -bash-4.1$ cd nm-local-dir/ > -bash-4.1$ ll * > filecache: > total 0 > > nmPrivate: > total 0 > > usercache: > total 0 > > I'm using Open JDK, would that be a problem? > > More log: > > STARTUP_MSG: java = 1.7.0_75 > ************************************************************/ > 2015-04-19 14:38:58,168 INFO > org.apache.hadoop.yarn.server.nodemanager.NodeManager: registered UNIX > signal handlers for [TERM, HUP, INT] > 2015-04-19 14:38:58,562 WARN org.apache.hadoop.util.NativeCodeLoader: > Unable to load native-hadoop library for your platform... using > builtin-java classes where applicable > 2015-04-19 14:38:59,018 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: > Registering class > org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType > for class > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher > 2015-04-19 14:38:59,020 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: > Registering class > org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationEventType > for class > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher > 2015-04-19 14:38:59,021 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: > Registering class > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizationEventType > for class > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService > 2015-04-19 14:38:59,021 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: > Registering class > org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServicesEventType > for class > org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices > 2015-04-19 14:38:59,022 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: > Registering class > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorEventType > for class > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl > 2015-04-19 14:38:59,023 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: > Registering class > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEventType > for class > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher > 2015-04-19 14:38:59,054 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: > Registering class > org.apache.hadoop.yarn.server.nodemanager.ContainerManagerEventType for > class > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl > 2015-04-19 14:38:59,054 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: > Registering class > org.apache.hadoop.yarn.server.nodemanager.NodeManagerEventType for class > org.apache.hadoop.yarn.server.nodemanager.NodeManager > 2015-04-19 14:38:59,109 INFO > org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from > hadoop-metrics2.properties > 2015-04-19 14:38:59,197 INFO > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot > period at 10 second(s). > 2015-04-19 14:38:59,197 INFO > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics > system started > 2015-04-19 14:38:59,217 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: > Registering class > org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.event.LogHandlerEventType > for class > org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler > 2015-04-19 14:38:59,217 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: > per directory file limit = 8192 > 2015-04-19 14:38:59,227 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: > Registering class > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizerEventType > for class > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker > 2015-04-19 14:38:59,248 WARN > org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The > Auxilurary Service named 'mapreduce_shuffle' in the configuration is for > class class org.apache.hadoop.mapred.ShuffleHandler which has a name of > 'httpshuffle'. Because these are not the same tools trying to send > ServiceData and read Service Meta Data may have issues unless the refer to > the name in the config. > 2015-04-19 14:38:59,248 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: > Adding auxiliary service httpshuffle, "mapreduce_shuffle" > 2015-04-19 14:38:59,281 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > Using ResourceCalculatorPlugin : > org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@7fc514a7 > 2015-04-19 14:38:59,281 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > Using ResourceCalculatorProcessTree : null > 2015-04-19 14:38:59,281 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > Physical memory check enabled: true > 2015-04-19 14:38:59,281 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > Virtual memory check enabled: true > 2015-04-19 14:38:59,284 WARN > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > NodeManager configured with 14 G physical memory allocated to containers, > which is more than 80% of the total physical memory available (14.7 G). > Thrashing might happen. > 2015-04-19 14:38:59,287 INFO > org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: > Initialized nodemanager for null: physical-memory=14336 > virtual-memory=30106 virtual-cores=8 > 2015-04-19 14:38:59,318 INFO org.apache.hadoop.ipc.CallQueueManager: Using > callQueue class java.util.concurrent.LinkedBlockingQueue > 2015-04-19 14:38:59,334 INFO org.apache.hadoop.ipc.Server: Starting Socket > Reader #1 for port 38230 > 2015-04-19 14:38:59,359 INFO > org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding > protocol org.apache.hadoop.yarn.api.ContainerManagementProtocolPB to the > server > 2015-04-19 14:38:59,359 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: > Blocking new container-requests as container manager rpc server is still > starting. > 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server > Responder: starting > 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server > listener on 38230: starting > 2015-04-19 14:38:59,366 INFO > org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager: > Updating node address : ip-10-100-70-199.ec2.internal:38230 > 2015-04-19 14:38:59,372 INFO org.apache.hadoop.ipc.CallQueueManager: Using > callQueue class java.util.concurrent.LinkedBlockingQueue > 2015-04-19 14:38:59,373 INFO org.apache.hadoop.ipc.Server: Starting Socket > Reader #1 for port 8040 > 2015-04-19 14:38:59,376 INFO > org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding > protocol > org.apache.hadoop.yarn.server.nodemanager.api.LocalizationProtocolPB to the > server > 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server > Responder: starting > 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server > listener on 8040: starting > 2015-04-19 14:38:59,380 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: > Localizer started on port 8040 > 2015-04-19 14:38:59,391 INFO org.apache.hadoop.mapred.IndexCache: > IndexCache created with max memory = 10485760 > 2015-04-19 14:38:59,403 INFO org.apache.hadoop.mapred.ShuffleHandler: > httpshuffle listening on port 13562 > 2015-04-19 14:38:59,405 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: > ContainerManager started at datanode-03.prod.com/10.100.70.199:38230 > 2015-04-19 14:38:59,405 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: > ContainerManager bound to 0.0.0.0/0.0.0.0:0 > 2015-04-19 14:38:59,405 INFO > org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating > NMWebApp at 0.0.0.0:8042 > 2015-04-19 14:38:59,471 INFO org.mortbay.log: Logging to > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via > org.mortbay.log.Slf4jLog > 2015-04-19 14:38:59,475 INFO org.apache.hadoop.http.HttpRequestLog: Http > request log for http.requests.nodemanager is not defined > 2015-04-19 14:38:59,487 INFO org.apache.hadoop.http.HttpServer2: Added > global filter 'safety' > (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) > 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added > filter static_user_filter > (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to > context node > 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added > filter static_user_filter > (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to > context logs > 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added > filter static_user_filter > (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to > context static > 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding > path spec: /node/* > 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding > path spec: /ws/* > 2015-04-19 14:38:59,505 INFO org.apache.hadoop.http.HttpServer2: Jetty > bound to port 8042 > 2015-04-19 14:38:59,505 INFO org.mortbay.log: jetty-6.1.26 > 2015-04-19 14:38:59,545 INFO org.mortbay.log: Extract > jar:file:/opt/test/service/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.6.0.jar!/webapps/node > to /tmp/Jetty_0_0_0_0_8042_node____19tj0x/webapp > 2015-04-19 14:38:59,778 INFO org.mortbay.log: Started HttpServer2$ > [email protected]:8042 > 2015-04-19 14:38:59,778 INFO org.apache.hadoop.yarn.webapp.WebApps: Web > app /node started at 8042 > 2015-04-19 14:39:00,093 INFO org.apache.hadoop.yarn.webapp.WebApps: > Registered webapp guice modules > 2015-04-19 14:39:00,126 INFO > org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending > out 0 NM container statuses: [] > 2015-04-19 14:39:00,131 INFO > org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: > Registering with RM using containers :[] > 2015-04-19 14:39:00,176 INFO > org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager: > Rolling master-key for container-tokens, got key with id -543066608 > 2015-04-19 14:39:00,178 INFO > org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM: > Rolling master-key for container-tokens, got key with id -1243797706 > 2015-04-19 14:39:00,179 INFO > org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered > with ResourceManager as ip-10-100-70-199.ec2.internal:38230 with total > resource of <memory:14336, vCores:8> > 2015-04-19 14:39:00,179 INFO > org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying > ContainerManager to unblock new container-requests > 2015-04-19 19:22:17,729 INFO SecurityLogger.org.apache.hadoop.ipc.Server: > Auth successful for appattempt_1429450734039_0010_000001 (auth:SIMPLE) > 2015-04-19 19:22:17,807 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: > Start request for container_1429450734039_0010_01_000001 by user nobody > 2015-04-19 19:22:17,828 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: > Creating a new application reference for app application_1429450734039_0010 > 2015-04-19 19:22:17,834 INFO > org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody > IP=10.100.66.251 OPERATION=Start Container Request > TARGET=ContainerManageImpl RESULT=SUCCESS > APPID=application_1429450734039_0010 > CONTAINERID=container_1429450734039_0010_01_000001 > 2015-04-19 19:22:17,835 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: > Application application_1429450734039_0010 transitioned from NEW to INITING > 2015-04-19 19:22:17,835 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: > Adding container_1429450734039_0010_01_000001 to application > application_1429450734039_0010 > 2015-04-19 19:22:17,839 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: > Application application_1429450734039_0010 transitioned from INITING to > RUNNING > 2015-04-19 19:22:17,843 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: > Container container_1429450734039_0010_01_000001 transitioned from NEW to > LOCALIZING > 2015-04-19 19:22:17,843 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got > event CONTAINER_INIT for appId application_1429450734039_0010 > 2015-04-19 19:22:17,876 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: > Resource > hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.jar > transitioned from INIT to DOWNLOADING > 2015-04-19 19:22:17,877 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: > Resource > hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.splitmetainfo > transitioned from INIT to DOWNLOADING > 2015-04-19 19:22:17,877 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: > Resource > hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.split > transitioned from INIT to DOWNLOADING > 2015-04-19 19:22:17,877 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: > Resource > hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.xml > transitioned from INIT to DOWNLOADING > 2015-04-19 19:22:17,877 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: > Created localizer for container_1429450734039_0010_01_000001 > 2015-04-19 19:22:17,880 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: > Localizer failed > java.lang.NullPointerException > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268) > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115) > at > org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075) > 2015-04-19 19:22:17,882 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: > Container container_1429450734039_0010_01_000001 transitioned from > LOCALIZING to LOCALIZATION_FAILED > 2015-04-19 19:22:17,886 WARN > org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody > OPERATION=Container > Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container > failed with state: LOCALIZATION_FAILED > APPID=application_1429450734039_0010 > CONTAINERID=container_1429450734039_0010_01_000001 > 2015-04-19 19:22:17,889 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: > Container container_1429450734039_0010_01_000001 transitioned from > LOCALIZATION_FAILED to DONE > 2015-04-19 19:22:17,889 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: > Removing container_1429450734039_0010_01_000001 from application > application_1429450734039_0010 > > On Sun, Apr 19, 2015 at 1:16 PM, Brahma Reddy Battula < > [email protected]> wrote: > >> As Alexander Alten-Lorenz pointed, it mostly config >> issue(yarn.nodemanager.local-dirs >> or mapred.local.dir).. >> >> can you able provide full logs..? >> >> Bytheway NPE is handled in Trunk ..Please check HADOOP-8436 for more >> details.. >> >> ------------------------------ >> From: [email protected] >> Subject: Re: ResourceLocalizationService: Localizer failed when running >> pi example >> Date: Sun, 19 Apr 2015 17:59:13 +0200 >> To: [email protected] >> >> >> As you said, that looks like a config issue. I would spot on the NM's >> local scratch dir (yarn.nodemanager.local-dirs). >> >> But without a complete stack trace, its a blind call. >> >> BR, >> AL >> >> -- >> mapredit.blogspot.com >> >> On Apr 18, 2015, at 6:24 PM, Fernando O. <[email protected]> wrote: >> >> Hey All, >> It's me again with another noob question: I deployed a cluster (HA >> mode) everything looked good but when I tried to run the pi example: >> >> bin/hadoop jar >> ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100 >> >> the same error occurs if I try to generate data with teragen 100000000 >> /test/data >> >> >> 2015-04-18 15:49:04,090 INFO >> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: >> Localizer failed >> java.lang.NullPointerException >> at >> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268) >> at >> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344) >> at >> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) >> at >> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131) >> at >> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115) >> at >> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420) >> at >> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075) >> >> >> I'm guessing it's a configuration issue but I don't know what am I >> missing :S >> >> >> >
