Make sure your mapred.local.dir (check it in mapred-site.xml) is actually exists and writable by your mapreduce usewr.
Thank you! Sincerely, Leonid Fedotov On Jul 9, 2013, at 6:09 PM, Kiran Dangeti wrote: > Hi Siddharth, > > While running the multi-node we need to take care of the local host of the > slave machine from the error messages the task tracker root directory not > able to get to the masters. Please check and rerun it. > > Thanks, > Kiran > > > On Tue, Jul 9, 2013 at 10:26 PM, siddharth mathur <[email protected]> wrote: > Hi, > > I have installed Hadoop 1.1.2 on a 5 nodes cluster. I installed it watching > this tutorial > http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/ > > > When I startup the hadoop, I get the folloing error in all the tasktrackers. > > " > 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > job_201307051203_0001 for user-log deletion with retainTimeStamp:1373472921775 > 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > job_201307051611_0001 for user-log deletion with retainTimeStamp:1373472921775 > 2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker: Failed to > get system directory... > 2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed to > get system directory... > 2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed to > get system directory... > 2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed to > get system directory... > " > > But everything looks fine in the webUI. > > When I run a job, I get the following error but the job completes anyways. I > have attached the screenshots of the maptask failed error log in the UI. > > " > 13/07/09 12:29:37 INFO input.FileInputFormat: Total input paths to process : 2 > 13/07/09 12:29:37 INFO util.NativeCodeLoader: Loaded the native-hadoop library > 13/07/09 12:29:37 WARN snappy.LoadSnappy: Snappy native library not loaded > 13/07/09 12:29:37 INFO mapred.JobClient: Running job: job_201307091215_0001 > 13/07/09 12:29:38 INFO mapred.JobClient: map 0% reduce 0% > 13/07/09 12:29:41 INFO mapred.JobClient: Task Id : > attempt_201307091215_0001_m_000001_0, Status : FAILED > Error initializing attempt_201307091215_0001_m_000001_0: > ENOENT: No such file or directory > at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method) > at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699) > at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654) > at > org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509) > at > org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344) > at > org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240) > at > org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205) > at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) > at > org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306) > at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221) > at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581) > at java.lang.Thread.run(Thread.java:724) > > 13/07/09 12:29:41 WARN mapred.JobClient: Error reading task > outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stdout > 13/07/09 12:29:41 WARN mapred.JobClient: Error reading task > outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stderr > 13/07/09 12:29:45 INFO mapred.JobClient: map 50% reduce 0% > 13/07/09 12:29:53 INFO mapred.JobClient: map 50% reduce 16% > 13/07/09 12:30:38 INFO mapred.JobClient: Task Id : > attempt_201307091215_0001_m_000000_1, Status : FAILED > Error initializing attempt_201307091215_0001_m_000000_1: > ENOENT: No such file or directory > at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method) > at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699) > at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654) > at > org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509) > at > org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344) > at > org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240) > at > org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205) > at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) > at > org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306) > at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221) > at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581) > at java.lang.Thread.run(Thread.java:724) > > 13/07/09 12:30:38 WARN mapred.JobClient: Error reading task > outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stdout > 13/07/09 12:30:38 WARN mapred.JobClient: Error reading task > outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stderr > 13/07/09 12:30:44 INFO mapred.JobClient: map 100% reduce 16% > 13/07/09 12:30:50 INFO mapred.JobClient: map 100% reduce 100% > 13/07/09 12:30:52 INFO mapred.JobClient: Job complete: job_201307091215_0001 > 13/07/09 12:30:52 INFO mapred.JobClient: Counters: 30 > 13/07/09 12:30:52 INFO mapred.JobClient: Job Counters > 13/07/09 12:30:52 INFO mapred.JobClient: Launched reduce tasks=1 > 13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=68494 > 13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all reduces > waiting after reserving slots (ms)=0 > 13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all maps > waiting after reserving slots (ms)=0 > 13/07/09 12:30:52 INFO mapred.JobClient: Rack-local map tasks=1 > 13/07/09 12:30:52 INFO mapred.JobClient: Launched map tasks=5 > 13/07/09 12:30:52 INFO mapred.JobClient: Data-local map tasks=4 > 13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=64711 > 13/07/09 12:30:52 INFO mapred.JobClient: File Output Format Counters > 13/07/09 12:30:52 INFO mapred.JobClient: Bytes Written=51 > 13/07/09 12:30:52 INFO mapred.JobClient: FileSystemCounters > 13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_READ=111 > 13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_READ=271 > 13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_WRITTEN=153612 > 13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=51 > 13/07/09 12:30:52 INFO mapred.JobClient: File Input Format Counters > 13/07/09 12:30:52 INFO mapred.JobClient: Bytes Read=45 > 13/07/09 12:30:52 INFO mapred.JobClient: Map-Reduce Framework > 13/07/09 12:30:52 INFO mapred.JobClient: Map output materialized bytes=117 > 13/07/09 12:30:52 INFO mapred.JobClient: Map input records=2 > 13/07/09 12:30:52 INFO mapred.JobClient: Reduce shuffle bytes=117 > 13/07/09 12:30:52 INFO mapred.JobClient: Spilled Records=20 > 13/07/09 12:30:52 INFO mapred.JobClient: Map output bytes=85 > 13/07/09 12:30:52 INFO mapred.JobClient: Total committed heap usage > (bytes)=602996736 > 13/07/09 12:30:52 INFO mapred.JobClient: CPU time spent (ms)=2560 > 13/07/09 12:30:52 INFO mapred.JobClient: Combine input records=10 > 13/07/09 12:30:52 INFO mapred.JobClient: SPLIT_RAW_BYTES=226 > 13/07/09 12:30:52 INFO mapred.JobClient: Reduce input records=10 > 13/07/09 12:30:52 INFO mapred.JobClient: Reduce input groups=8 > 13/07/09 12:30:52 INFO mapred.JobClient: Combine output records=10 > 13/07/09 12:30:52 INFO mapred.JobClient: Physical memory (bytes) > snapshot=520691712 > 13/07/09 12:30:52 INFO mapred.JobClient: Reduce output records=8 > 13/07/09 12:30:52 INFO mapred.JobClient: Virtual memory (bytes) > snapshot=3025334272 > 13/07/09 12:30:52 INFO mapred.JobClient: Map output records=10 > > " > > > Following is the complete tasktracker log on one node after starting the > hadoop (not running the job): > > " > 2013-07-09 12:15:15,426 INFO org.apache.hadoop.mapred.TaskTracker: > STARTUP_MSG: > /************************************************************ > STARTUP_MSG: Starting TaskTracker > STARTUP_MSG: host = dmkd-1/172.31.21.10 > STARTUP_MSG: args = [] > STARTUP_MSG: version = 1.1.2 > STARTUP_MSG: build = > https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1 -r > 1440782; compiled by 'hortonfo' on Thu Jan 31 02:03:24 UTC 2013 > ************************************************************/ > 2013-07-09 12:15:15,722 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: > loaded properties from hadoop-metrics2.properties > 2013-07-09 12:15:15,823 INFO > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source > MetricsSystem,sub=Stats registered. > 2013-07-09 12:15:15,824 INFO > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period > at 10 second(s). > 2013-07-09 12:15:15,824 INFO > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics system > started > 2013-07-09 12:15:16,089 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded > the native-hadoop library > 2013-07-09 12:15:16,223 INFO > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi > registered. > 2013-07-09 12:15:16,230 WARN > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already > exists! > 2013-07-09 12:15:21,528 INFO org.mortbay.log: Logging to > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via > org.mortbay.log.Slf4jLog > 2013-07-09 12:15:21,600 INFO org.apache.hadoop.http.HttpServer: Added global > filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) > 2013-07-09 12:15:21,631 INFO org.apache.hadoop.mapred.TaskLogsTruncater: > Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1 > 2013-07-09 12:15:21,636 INFO org.apache.hadoop.mapred.TaskTracker: Starting > tasktracker with owner as hduser > 2013-07-09 12:15:21,637 INFO org.apache.hadoop.mapred.TaskTracker: Good > mapred local directories are: /app/hadoop/tmp/mapred/local > 2013-07-09 12:15:21,652 INFO > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm > registered. > 2013-07-09 12:15:21,653 INFO > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source > TaskTrackerMetrics registered. > 2013-07-09 12:15:21,680 INFO org.apache.hadoop.ipc.Server: Starting > SocketReader > 2013-07-09 12:15:21,683 INFO > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source > RpcDetailedActivityForPort36783 registered. > 2013-07-09 12:15:21,684 INFO > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source > RpcActivityForPort36783 registered. > 2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server > Responder: starting > 2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server > listener on 36783: starting > 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server handler > 0 on 36783: starting > 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server handler > 1 on 36783: starting > 2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server handler > 2 on 36783: starting > 2013-07-09 12:15:21,692 INFO org.apache.hadoop.ipc.Server: IPC Server handler > 3 on 36783: starting > 2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker: > TaskTracker up at: 127.0.0.1/127.0.0.1:36783 > 2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker: Starting > tracker tracker_dmkd-1:127.0.0.1/127.0.0.1:36783 > 2013-07-09 12:15:21,750 INFO org.apache.hadoop.mapred.TaskTracker: Starting > thread: Map-events fetcher for all reduce tasks on > tracker_dmkd-1:127.0.0.1/127.0.0.1:36783 > 2013-07-09 12:15:21,762 INFO org.apache.hadoop.util.ProcessTree: setsid > exited with exit code 0 > 2013-07-09 12:15:21,766 INFO org.apache.hadoop.mapred.TaskTracker: Using > ResourceCalculatorPlugin : > org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7ec79676 > 2013-07-09 12:15:21,770 WARN org.apache.hadoop.mapred.TaskTracker: > TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is > disabled. > 2013-07-09 12:15:21,777 INFO org.apache.hadoop.mapred.IndexCache: IndexCache > created with max memory = 10485760 > 2013-07-09 12:15:21,785 INFO > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source > ShuffleServerMetrics registered. > 2013-07-09 12:15:21,788 INFO org.apache.hadoop.http.HttpServer: Port returned > by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening > the listener on 50060 > 2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer: > listener.getLocalPort() returned 50060 > webServer.getConnectors()[0].getLocalPort() returned 50060 > 2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer: Jetty bound > to port 50060 > 2013-07-09 12:15:21,789 INFO org.mortbay.log: jetty-6.1.26 > 2013-07-09 12:15:22,286 INFO org.mortbay.log: Started > [email protected]:50060 > 2013-07-09 12:15:22,287 INFO org.apache.hadoop.mapred.TaskTracker: > FILE_CACHE_SIZE for mapOutputServlet set to : 2000 > 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > job_201307032034_0003 for user-log deletion with retainTimeStamp:1373472921773 > 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > job_201307051627_0001 for user-log deletion with retainTimeStamp:1373472921773 > 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > job_201307051203_0002 for user-log deletion with retainTimeStamp:1373472921773 > 2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > job_201307051135_0001 for user-log deletion with retainTimeStamp:1373472921773 > 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > job_201307032034_0002 for user-log deletion with retainTimeStamp:1373472921773 > 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > job_201307051327_0001 for user-log deletion with retainTimeStamp:1373472921773 > 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > job_201307051135_0005 for user-log deletion with retainTimeStamp:1373472921773 > 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > job_201307050127_0001 for user-log deletion with retainTimeStamp:1373472921773 > 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > job_201307050127_0002 for user-log deletion with retainTimeStamp:1373472921773 > 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > job_201307051500_0001 for user-log deletion with retainTimeStamp:1373472921773 > 2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > job_201307051634_0001 for user-log deletion with retainTimeStamp:1373472921773 > 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > job_201307032034_0001 for user-log deletion with retainTimeStamp:1373472921773 > 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > job_201307051203_0001 for user-log deletion with retainTimeStamp:1373472921773 > 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > job_201307051611_0001 for user-log deletion with retainTimeStamp:1373472921773 > 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > job_201307032034_0003 for user-log deletion with retainTimeStamp:1373472921775 > 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > job_201307032034_0003 for user-log deletion with retainTimeStamp:1373472921775 > 2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > job_201307051627_0001 for user-log deletion with retainTimeStamp:1373472921775 > 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > job_201307051203_0002 for user-log deletion with retainTimeStamp:1373472921775 > 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > job_201307051135_0001 for user-log deletion with retainTimeStamp:1373472921775 > 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > job_201307032034_0002 for user-log deletion with retainTimeStamp:1373472921775 > 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > job_201307051327_0001 for user-log deletion with retainTimeStamp:1373472921775 > 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > job_201307051135_0005 for user-log deletion with retainTimeStamp:1373472921775 > 2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > job_201307050127_0001 for user-log deletion with retainTimeStamp:1373472921775 > 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > job_201307050127_0002 for user-log deletion with retainTimeStamp:1373472921775 > 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > job_201307051500_0001 for user-log deletion with retainTimeStamp:1373472921775 > 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > job_201307051634_0001 for user-log deletion with retainTimeStamp:1373472921775 > 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > job_201307032034_0001 for user-log deletion with retainTimeStamp:1373472921775 > 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > job_201307051203_0001 for user-log deletion with retainTimeStamp:1373472921775 > 2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > job_201307051611_0001 for user-log deletion with retainTimeStamp:1373472921775 > 2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker: Failed to > get system directory... > 2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed to > get system directory... > 2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed to > get system directory... > 2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed to > get system directory... > 2013-07-09 12:15:33,738 INFO org.apache.hadoop.mapred.TaskTracker: Failed to > get system directory... > 2013-07-09 12:15:35,802 INFO org.apache.hadoop.mapred.TaskTracker: Failed to > get system directory... > 2013-07-09 12:15:37,308 INFO org.apache.hadoop.mapred.TaskTracker: Failed to > get system directory... > 2013-07-09 12:15:41,620 INFO org.apache.hadoop.mapred.TaskTracker: Failed to > get system directory... > 2013-07-09 12:15:44,310 INFO org.apache.hadoop.mapred.TaskTracker: Failed to > get system directory... > 2013-07-09 12:15:48,939 INFO org.apache.hadoop.mapred.TaskTracker: Failed to > get system directory... > 2013-07-09 12:15:50,121 INFO org.apache.hadoop.mapred.TaskTracker: Failed to > get system directory... > 2013-07-09 12:15:52,549 INFO org.apache.hadoop.mapred.TaskTracker: Failed to > get system directory... > 2013-07-09 12:15:53,751 INFO org.apache.hadoop.mapred.TaskTracker: Failed to > get system directory... > 2013-07-09 12:15:56,464 INFO org.apache.hadoop.mapred.TaskTracker: Failed to > get system directory... > 2013-07-09 12:15:58,423 INFO org.apache.hadoop.mapred.TaskTracker: Failed to > get system directory... > 2013-07-09 12:16:03,413 INFO org.apache.hadoop.mapred.TaskTracker: Failed to > get system directory... > 2013-07-09 12:16:04,567 INFO org.apache.hadoop.mapred.TaskTracker: Failed to > get system directory... > 2013-07-09 12:16:08,177 INFO org.apache.hadoop.mapred.TaskTracker: Failed to > get system directory... > > " > > Please help. > > > Thanks and Regards, > Siddharth > > >
