I didn't specify it so it's using the default value (in /tmp) On Sun, Apr 19, 2015 at 10:21 PM, Drake민영근 <[email protected]> wrote:
> Hi, > > guess the "yarn.nodemanager.local-dirs" property is the problem. Can you > provide that part of yarn-site.xml? > > Thanks. > > Drake 민영근 Ph.D > kt NexR > > On Mon, Apr 20, 2015 at 4:27 AM, Fernando O. <[email protected]> wrote: > >> yeah... there's not much there: >> >> -bash-4.1$ cd nm-local-dir/ >> -bash-4.1$ ll * >> filecache: >> total 0 >> >> nmPrivate: >> total 0 >> >> usercache: >> total 0 >> >> I'm using Open JDK, would that be a problem? >> >> More log: >> >> STARTUP_MSG: java = 1.7.0_75 >> ************************************************************/ >> 2015-04-19 14:38:58,168 INFO >> org.apache.hadoop.yarn.server.nodemanager.NodeManager: registered UNIX >> signal handlers for [TERM, HUP, INT] >> 2015-04-19 14:38:58,562 WARN org.apache.hadoop.util.NativeCodeLoader: >> Unable to load native-hadoop library for your platform... using >> builtin-java classes where applicable >> 2015-04-19 14:38:59,018 INFO >> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class >> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType >> for class >> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher >> 2015-04-19 14:38:59,020 INFO >> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class >> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationEventType >> for class >> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher >> 2015-04-19 14:38:59,021 INFO >> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class >> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizationEventType >> for class >> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService >> 2015-04-19 14:38:59,021 INFO >> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class >> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServicesEventType >> for class >> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices >> 2015-04-19 14:38:59,022 INFO >> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class >> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorEventType >> for class >> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl >> 2015-04-19 14:38:59,023 INFO >> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class >> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEventType >> for class >> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher >> 2015-04-19 14:38:59,054 INFO >> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class >> org.apache.hadoop.yarn.server.nodemanager.ContainerManagerEventType for >> class >> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl >> 2015-04-19 14:38:59,054 INFO >> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class >> org.apache.hadoop.yarn.server.nodemanager.NodeManagerEventType for class >> org.apache.hadoop.yarn.server.nodemanager.NodeManager >> 2015-04-19 14:38:59,109 INFO >> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from >> hadoop-metrics2.properties >> 2015-04-19 14:38:59,197 INFO >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot >> period at 10 second(s). >> 2015-04-19 14:38:59,197 INFO >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics >> system started >> 2015-04-19 14:38:59,217 INFO >> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class >> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.event.LogHandlerEventType >> for class >> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler >> 2015-04-19 14:38:59,217 INFO >> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: >> per directory file limit = 8192 >> 2015-04-19 14:38:59,227 INFO >> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class >> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizerEventType >> for class >> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker >> 2015-04-19 14:38:59,248 WARN >> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The >> Auxilurary Service named 'mapreduce_shuffle' in the configuration is for >> class class org.apache.hadoop.mapred.ShuffleHandler which has a name of >> 'httpshuffle'. Because these are not the same tools trying to send >> ServiceData and read Service Meta Data may have issues unless the refer to >> the name in the config. >> 2015-04-19 14:38:59,248 INFO >> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: >> Adding auxiliary service httpshuffle, "mapreduce_shuffle" >> 2015-04-19 14:38:59,281 INFO >> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: >> Using ResourceCalculatorPlugin : >> org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@7fc514a7 >> 2015-04-19 14:38:59,281 INFO >> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: >> Using ResourceCalculatorProcessTree : null >> 2015-04-19 14:38:59,281 INFO >> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: >> Physical memory check enabled: true >> 2015-04-19 14:38:59,281 INFO >> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: >> Virtual memory check enabled: true >> 2015-04-19 14:38:59,284 WARN >> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: >> NodeManager configured with 14 G physical memory allocated to containers, >> which is more than 80% of the total physical memory available (14.7 G). >> Thrashing might happen. >> 2015-04-19 14:38:59,287 INFO >> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: >> Initialized nodemanager for null: physical-memory=14336 >> virtual-memory=30106 virtual-cores=8 >> 2015-04-19 14:38:59,318 INFO org.apache.hadoop.ipc.CallQueueManager: >> Using callQueue class java.util.concurrent.LinkedBlockingQueue >> 2015-04-19 14:38:59,334 INFO org.apache.hadoop.ipc.Server: Starting >> Socket Reader #1 for port 38230 >> 2015-04-19 14:38:59,359 INFO >> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding >> protocol org.apache.hadoop.yarn.api.ContainerManagementProtocolPB to the >> server >> 2015-04-19 14:38:59,359 INFO >> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: >> Blocking new container-requests as container manager rpc server is still >> starting. >> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server >> Responder: starting >> 2015-04-19 14:38:59,359 INFO org.apache.hadoop.ipc.Server: IPC Server >> listener on 38230: starting >> 2015-04-19 14:38:59,366 INFO >> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager: >> Updating node address : ip-10-100-70-199.ec2.internal:38230 >> 2015-04-19 14:38:59,372 INFO org.apache.hadoop.ipc.CallQueueManager: >> Using callQueue class java.util.concurrent.LinkedBlockingQueue >> 2015-04-19 14:38:59,373 INFO org.apache.hadoop.ipc.Server: Starting >> Socket Reader #1 for port 8040 >> 2015-04-19 14:38:59,376 INFO >> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding >> protocol >> org.apache.hadoop.yarn.server.nodemanager.api.LocalizationProtocolPB to the >> server >> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server >> Responder: starting >> 2015-04-19 14:38:59,376 INFO org.apache.hadoop.ipc.Server: IPC Server >> listener on 8040: starting >> 2015-04-19 14:38:59,380 INFO >> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: >> Localizer started on port 8040 >> 2015-04-19 14:38:59,391 INFO org.apache.hadoop.mapred.IndexCache: >> IndexCache created with max memory = 10485760 >> 2015-04-19 14:38:59,403 INFO org.apache.hadoop.mapred.ShuffleHandler: >> httpshuffle listening on port 13562 >> 2015-04-19 14:38:59,405 INFO >> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: >> ContainerManager started at datanode-03.prod.com/10.100.70.199:38230 >> 2015-04-19 14:38:59,405 INFO >> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: >> ContainerManager bound to 0.0.0.0/0.0.0.0:0 >> 2015-04-19 14:38:59,405 INFO >> org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating >> NMWebApp at 0.0.0.0:8042 >> 2015-04-19 14:38:59,471 INFO org.mortbay.log: Logging to >> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via >> org.mortbay.log.Slf4jLog >> 2015-04-19 14:38:59,475 INFO org.apache.hadoop.http.HttpRequestLog: Http >> request log for http.requests.nodemanager is not defined >> 2015-04-19 14:38:59,487 INFO org.apache.hadoop.http.HttpServer2: Added >> global filter 'safety' >> (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) >> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added >> filter static_user_filter >> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to >> context node >> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added >> filter static_user_filter >> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to >> context logs >> 2015-04-19 14:38:59,489 INFO org.apache.hadoop.http.HttpServer2: Added >> filter static_user_filter >> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to >> context static >> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding >> path spec: /node/* >> 2015-04-19 14:38:59,493 INFO org.apache.hadoop.http.HttpServer2: adding >> path spec: /ws/* >> 2015-04-19 14:38:59,505 INFO org.apache.hadoop.http.HttpServer2: Jetty >> bound to port 8042 >> 2015-04-19 14:38:59,505 INFO org.mortbay.log: jetty-6.1.26 >> 2015-04-19 14:38:59,545 INFO org.mortbay.log: Extract >> jar:file:/opt/test/service/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.6.0.jar!/webapps/node >> to /tmp/Jetty_0_0_0_0_8042_node____19tj0x/webapp >> 2015-04-19 14:38:59,778 INFO org.mortbay.log: Started HttpServer2$ >> [email protected]:8042 >> 2015-04-19 14:38:59,778 INFO org.apache.hadoop.yarn.webapp.WebApps: Web >> app /node started at 8042 >> 2015-04-19 14:39:00,093 INFO org.apache.hadoop.yarn.webapp.WebApps: >> Registered webapp guice modules >> 2015-04-19 14:39:00,126 INFO >> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending >> out 0 NM container statuses: [] >> 2015-04-19 14:39:00,131 INFO >> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: >> Registering with RM using containers :[] >> 2015-04-19 14:39:00,176 INFO >> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager: >> Rolling master-key for container-tokens, got key with id -543066608 >> 2015-04-19 14:39:00,178 INFO >> org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM: >> Rolling master-key for container-tokens, got key with id -1243797706 >> 2015-04-19 14:39:00,179 INFO >> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered >> with ResourceManager as ip-10-100-70-199.ec2.internal:38230 with total >> resource of <memory:14336, vCores:8> >> 2015-04-19 14:39:00,179 INFO >> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying >> ContainerManager to unblock new container-requests >> 2015-04-19 19:22:17,729 INFO SecurityLogger.org.apache.hadoop.ipc.Server: >> Auth successful for appattempt_1429450734039_0010_000001 (auth:SIMPLE) >> 2015-04-19 19:22:17,807 INFO >> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: >> Start request for container_1429450734039_0010_01_000001 by user nobody >> 2015-04-19 19:22:17,828 INFO >> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: >> Creating a new application reference for app application_1429450734039_0010 >> 2015-04-19 19:22:17,834 INFO >> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody >> IP=10.100.66.251 OPERATION=Start Container Request >> TARGET=ContainerManageImpl RESULT=SUCCESS >> APPID=application_1429450734039_0010 >> CONTAINERID=container_1429450734039_0010_01_000001 >> 2015-04-19 19:22:17,835 INFO >> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: >> Application application_1429450734039_0010 transitioned from NEW to INITING >> 2015-04-19 19:22:17,835 INFO >> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: >> Adding container_1429450734039_0010_01_000001 to application >> application_1429450734039_0010 >> 2015-04-19 19:22:17,839 INFO >> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: >> Application application_1429450734039_0010 transitioned from INITING to >> RUNNING >> 2015-04-19 19:22:17,843 INFO >> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: >> Container container_1429450734039_0010_01_000001 transitioned from NEW to >> LOCALIZING >> 2015-04-19 19:22:17,843 INFO >> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got >> event CONTAINER_INIT for appId application_1429450734039_0010 >> 2015-04-19 19:22:17,876 INFO >> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: >> Resource >> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.jar >> transitioned from INIT to DOWNLOADING >> 2015-04-19 19:22:17,877 INFO >> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: >> Resource >> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.splitmetainfo >> transitioned from INIT to DOWNLOADING >> 2015-04-19 19:22:17,877 INFO >> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: >> Resource >> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.split >> transitioned from INIT to DOWNLOADING >> 2015-04-19 19:22:17,877 INFO >> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: >> Resource >> hdfs://rdcluster:8020/tmp/hadoop-yarn/staging/nobody/.staging/job_1429450734039_0010/job.xml >> transitioned from INIT to DOWNLOADING >> 2015-04-19 19:22:17,877 INFO >> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: >> Created localizer for container_1429450734039_0010_01_000001 >> 2015-04-19 19:22:17,880 INFO >> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: >> Localizer failed >> java.lang.NullPointerException >> at >> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268) >> at >> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344) >> at >> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) >> at >> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131) >> at >> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115) >> at >> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420) >> at >> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075) >> 2015-04-19 19:22:17,882 INFO >> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: >> Container container_1429450734039_0010_01_000001 transitioned from >> LOCALIZING to LOCALIZATION_FAILED >> 2015-04-19 19:22:17,886 WARN >> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=nobody >> OPERATION=Container >> Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container >> failed with state: LOCALIZATION_FAILED >> APPID=application_1429450734039_0010 >> CONTAINERID=container_1429450734039_0010_01_000001 >> 2015-04-19 19:22:17,889 INFO >> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: >> Container container_1429450734039_0010_01_000001 transitioned from >> LOCALIZATION_FAILED to DONE >> 2015-04-19 19:22:17,889 INFO >> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: >> Removing container_1429450734039_0010_01_000001 from application >> application_1429450734039_0010 >> >> On Sun, Apr 19, 2015 at 1:16 PM, Brahma Reddy Battula < >> [email protected]> wrote: >> >>> As Alexander Alten-Lorenz pointed, it mostly config >>> issue(yarn.nodemanager.local-dirs >>> or mapred.local.dir).. >>> >>> can you able provide full logs..? >>> >>> Bytheway NPE is handled in Trunk ..Please check HADOOP-8436 for more >>> details.. >>> >>> ------------------------------ >>> From: [email protected] >>> Subject: Re: ResourceLocalizationService: Localizer failed when running >>> pi example >>> Date: Sun, 19 Apr 2015 17:59:13 +0200 >>> To: [email protected] >>> >>> >>> As you said, that looks like a config issue. I would spot on the NM's >>> local scratch dir (yarn.nodemanager.local-dirs). >>> >>> But without a complete stack trace, its a blind call. >>> >>> BR, >>> AL >>> >>> -- >>> mapredit.blogspot.com >>> >>> On Apr 18, 2015, at 6:24 PM, Fernando O. <[email protected]> wrote: >>> >>> Hey All, >>> It's me again with another noob question: I deployed a cluster (HA >>> mode) everything looked good but when I tried to run the pi example: >>> >>> bin/hadoop jar >>> ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 16 100 >>> >>> the same error occurs if I try to generate data with teragen 100000000 >>> /test/data >>> >>> >>> 2015-04-18 15:49:04,090 INFO >>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: >>> Localizer failed >>> java.lang.NullPointerException >>> at >>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268) >>> at >>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344) >>> at >>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) >>> at >>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131) >>> at >>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115) >>> at >>> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420) >>> at >>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075) >>> >>> >>> I'm guessing it's a configuration issue but I don't know what am I >>> missing :S >>> >>> >>> >> >
