Siddharth, The error msgs pointing to file system issues. Make sure that the file system locations you specified in the config files are accurate and accessible.
-Sreedhar >________________________________ > From: siddharth mathur <[email protected]> >To: [email protected] >Sent: Tuesday, July 9, 2013 9:56 AM >Subject: Issues Running Hadoop 1.1.2 on multi-node cluster > > > >Hi, > >I have installed Hadoop 1.1.2 on a 5 nodes cluster. I installed it watching >this tutorial >http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/ > > >When I startup the hadoop, I get the folloing error in all the tasktrackers. > > >" >2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding >job_201307051203_0001 for user-log deletion with retainTimeStamp:1373472921775 >2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding >job_201307051611_0001 for user-log deletion with retainTimeStamp:1373472921775 >2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker:Failed to >get system directory... >2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed to >get system directory... >2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed to >get system directory... >2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed to >get system directory... >" > >But everything looks fine in the webUI. > > >When I run a job, I get the following error but the job completes anyways. I >haveattached the screenshots of the maptask failed error log in the UI. > >" > >13/07/09 12:29:37 INFO input.FileInputFormat: Total input paths to process : 2 >13/07/09 12:29:37 INFO util.NativeCodeLoader: Loaded the native-hadoop library >13/07/09 12:29:37 WARN snappy.LoadSnappy: Snappy native library not loaded >13/07/09 12:29:37 INFO mapred.JobClient: Running job: job_201307091215_0001 >13/07/09 12:29:38 INFO mapred.JobClient: map 0% reduce 0% >13/07/09 12:29:41 INFO mapred.JobClient: Task Id : >attempt_201307091215_0001_m_000001_0, Status : FAILED >Error initializing attempt_201307091215_0001_m_000001_0: >ENOENT: No such file or directory > at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method) > at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699) > at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654) > at >org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509) > at >org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344) > at >org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240) > at >org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205) > at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at >org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) > at >org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306) > at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221) > at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581) > at java.lang.Thread.run(Thread.java:724) > >13/07/09 12:29:41 WARN mapred.JobClient: Error reading task >outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stdout >13/07/09 12:29:41 WARN mapred.JobClient: Error reading task >outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stderr >13/07/09 12:29:45 INFO mapred.JobClient: map 50% reduce 0% >13/07/09 12:29:53 INFO mapred.JobClient: map 50% reduce 16% >13/07/09 12:30:38 INFO mapred.JobClient: Task Id : >attempt_201307091215_0001_m_000000_1, Status : FAILED >Error initializing attempt_201307091215_0001_m_000000_1: >ENOENT: No such file or directory > at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method) > at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699) > at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654) > at >org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509) > at >org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344) > at >org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240) > at >org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205) > at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at >org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) > at >org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306) > at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221) > at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581) > at java.lang.Thread.run(Thread.java:724) > >13/07/09 12:30:38 WARN mapred.JobClient: Error reading task >outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stdout >13/07/09 12:30:38 WARN mapred.JobClient: Error reading task >outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stderr >13/07/09 12:30:44 INFO mapred.JobClient: map 100% reduce 16% >13/07/09 12:30:50 INFO mapred.JobClient: map 100% reduce 100% >13/07/09 12:30:52 INFO mapred.JobClient: Job complete: job_201307091215_0001 >13/07/09 12:30:52 INFO mapred.JobClient: Counters: 30 >13/07/09 12:30:52 INFO mapred.JobClient: Job Counters >13/07/09 12:30:52 INFO mapred.JobClient: Launched reduce tasks=1 >13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=68494 >13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all reduces >waiting after reserving slots (ms)=0 >13/07/09 12:30:52 INFO mapred.JobClient: Total time spent by all maps >waiting after reserving slots (ms)=0 >13/07/09 12:30:52 INFO mapred.JobClient: Rack-local map tasks=1 >13/07/09 12:30:52 INFO mapred.JobClient: Launched map tasks=5 >13/07/09 12:30:52 INFO mapred.JobClient: Data-local map tasks=4 >13/07/09 12:30:52 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=64711 >13/07/09 12:30:52 INFO mapred.JobClient: File Output Format Counters >13/07/09 12:30:52 INFO mapred.JobClient: Bytes Written=51 >13/07/09 12:30:52 INFO mapred.JobClient: FileSystemCounters >13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_READ=111 >13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_READ=271 >13/07/09 12:30:52 INFO mapred.JobClient: FILE_BYTES_WRITTEN=153612 >13/07/09 12:30:52 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=51 >13/07/09 12:30:52 INFO mapred.JobClient: File Input Format Counters >13/07/09 12:30:52 INFO mapred.JobClient: Bytes Read=45 >13/07/09 12:30:52 INFO mapred.JobClient: Map-Reduce Framework >13/07/09 12:30:52 INFO mapred.JobClient: Map output materialized bytes=117 >13/07/09 12:30:52 INFO mapred.JobClient: Map input records=2 >13/07/09 12:30:52 INFO mapred.JobClient: Reduce shuffle bytes=117 >13/07/09 12:30:52 INFO mapred.JobClient: Spilled Records=20 >13/07/09 12:30:52 INFO mapred.JobClient: Map output bytes=85 >13/07/09 12:30:52 INFO mapred.JobClient: Total committed heap usage >(bytes)=602996736 >13/07/09 12:30:52 INFO mapred.JobClient: CPU time spent (ms)=2560 >13/07/09 12:30:52 INFO mapred.JobClient: Combine input records=10 >13/07/09 12:30:52 INFO mapred.JobClient: SPLIT_RAW_BYTES=226 >13/07/09 12:30:52 INFO mapred.JobClient: Reduce input records=10 >13/07/09 12:30:52 INFO mapred.JobClient: Reduce input groups=8 >13/07/09 12:30:52 INFO mapred.JobClient: Combine output records=10 >13/07/09 12:30:52 INFO mapred.JobClient: Physical memory (bytes) >snapshot=520691712 >13/07/09 12:30:52 INFO mapred.JobClient: Reduce output records=8 >13/07/09 12:30:52 INFO mapred.JobClient: Virtual memory (bytes) >snapshot=3025334272 >13/07/09 12:30:52 INFO mapred.JobClient: Map output records=10 > >" > > >Following is the complete tasktracker log on one node after starting the >hadoop (not running the job): > > >" > >2013-07-09 12:15:15,426 INFO org.apache.hadoop.mapred.TaskTracker: STARTUP_MSG: >/************************************************************ >STARTUP_MSG: Starting TaskTracker >STARTUP_MSG: host = dmkd-1/172.31.21.10 >STARTUP_MSG: args = [] >STARTUP_MSG: version = 1.1.2 >STARTUP_MSG: build = >https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1 -r 1440782; >compiled by 'hortonfo' on Thu Jan 31 02:03:24 UTC 2013 >************************************************************/ >2013-07-09 12:15:15,722 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: >loaded properties from hadoop-metrics2.properties >2013-07-09 12:15:15,823 INFO >org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source >MetricsSystem,sub=Stats registered. >2013-07-09 12:15:15,824 INFO >org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period >at 10 second(s). >2013-07-09 12:15:15,824 INFO >org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics system >started >2013-07-09 12:15:16,089 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded >the native-hadoop library >2013-07-09 12:15:16,223 INFO >org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi >registered. >2013-07-09 12:15:16,230 WARN >org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already >exists! >2013-07-09 12:15:21,528 INFO org.mortbay.log: Logging to >org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog >2013-07-09 12:15:21,600 INFO org.apache.hadoop.http.HttpServer: Added global >filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) >2013-07-09 12:15:21,631 INFO org.apache.hadoop.mapred.TaskLogsTruncater: >Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1 >2013-07-09 12:15:21,636 INFO org.apache.hadoop.mapred.TaskTracker: Starting >tasktracker with owner as hduser >2013-07-09 12:15:21,637 INFO org.apache.hadoop.mapred.TaskTracker: Good mapred >local directories are: /app/hadoop/tmp/mapred/local >2013-07-09 12:15:21,652 INFO >org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm >registered. >2013-07-09 12:15:21,653 INFO >org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source >TaskTrackerMetrics registered. >2013-07-09 12:15:21,680 INFO org.apache.hadoop.ipc.Server: Starting >SocketReader >2013-07-09 12:15:21,683 INFO >org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source >RpcDetailedActivityForPort36783 registered. >2013-07-09 12:15:21,684 INFO >org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source >RpcActivityForPort36783 registered. >2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server >Responder: starting >2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server listener >on 36783: starting >2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server handler >0 on 36783: starting >2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server handler >1 on 36783: starting >2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server handler >2 on 36783: starting >2013-07-09 12:15:21,692 INFO org.apache.hadoop.ipc.Server: IPC Server handler >3 on 36783: starting >2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker: TaskTracker >up at: 127.0.0.1/127.0.0.1:36783 >2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker: Starting >tracker tracker_dmkd-1:127.0.0.1/127.0.0.1:36783 >2013-07-09 12:15:21,750 INFO org.apache.hadoop.mapred.TaskTracker: Starting >thread: Map-events fetcher for all reduce tasks on >tracker_dmkd-1:127.0.0.1/127.0.0.1:36783 >2013-07-09 12:15:21,762 INFO org.apache.hadoop.util.ProcessTree: setsid exited >with exit code 0 >2013-07-09 12:15:21,766 INFO org.apache.hadoop.mapred.TaskTracker: Using >ResourceCalculatorPlugin : >org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7ec79676 >2013-07-09 12:15:21,770 WARN org.apache.hadoop.mapred.TaskTracker: >TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled. >2013-07-09 12:15:21,777 INFO org.apache.hadoop.mapred.IndexCache: IndexCache >created with max memory = 10485760 >2013-07-09 12:15:21,785 INFO >org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source >ShuffleServerMetrics registered. >2013-07-09 12:15:21,788 INFO org.apache.hadoop.http.HttpServer: Port returned >by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening >the listener on 50060 >2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer: >listener.getLocalPort() returned 50060 >webServer.getConnectors()[0].getLocalPort() returned 50060 >2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer: Jetty bound to >port 50060 >2013-07-09 12:15:21,789 INFO org.mortbay.log: jetty-6.1.26 >2013-07-09 12:15:22,286 INFO org.mortbay.log: Started >[email protected]:50060 >2013-07-09 12:15:22,287 INFO org.apache.hadoop.mapred.TaskTracker: >FILE_CACHE_SIZE for mapOutputServlet set to : 2000 >2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding >job_201307032034_0003 for user-log deletion with retainTimeStamp:1373472921773 >2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding >job_201307051627_0001 for user-log deletion with retainTimeStamp:1373472921773 >2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding >job_201307051203_0002 for user-log deletion with retainTimeStamp:1373472921773 >2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding >job_201307051135_0001 for user-log deletion with retainTimeStamp:1373472921773 >2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding >job_201307032034_0002 for user-log deletion with retainTimeStamp:1373472921773 >2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding >job_201307051327_0001 for user-log deletion with retainTimeStamp:1373472921773 >2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding >job_201307051135_0005 for user-log deletion with retainTimeStamp:1373472921773 >2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding >job_201307050127_0001 for user-log deletion with retainTimeStamp:1373472921773 >2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding >job_201307050127_0002 for user-log deletion with retainTimeStamp:1373472921773 >2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding >job_201307051500_0001 for user-log deletion with retainTimeStamp:1373472921773 >2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding >job_201307051634_0001 for user-log deletion with retainTimeStamp:1373472921773 >2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding >job_201307032034_0001 for user-log deletion with retainTimeStamp:1373472921773 >2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding >job_201307051203_0001 for user-log deletion with retainTimeStamp:1373472921773 >2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding >job_201307051611_0001 for user-log deletion with retainTimeStamp:1373472921773 >2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding >job_201307032034_0003 for user-log deletion with retainTimeStamp:1373472921775 >2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding >job_201307032034_0003 for user-log deletion with retainTimeStamp:1373472921775 >2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding >job_201307051627_0001 for user-log deletion with retainTimeStamp:1373472921775 >2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding >job_201307051203_0002 for user-log deletion with retainTimeStamp:1373472921775 >2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding >job_201307051135_0001 for user-log deletion with retainTimeStamp:1373472921775 >2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding >job_201307032034_0002 for user-log deletion with retainTimeStamp:1373472921775 >2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding >job_201307051327_0001 for user-log deletion with retainTimeStamp:1373472921775 >2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding >job_201307051135_0005 for user-log deletion with retainTimeStamp:1373472921775 >2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding >job_201307050127_0001 for user-log deletion with retainTimeStamp:1373472921775 >2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding >job_201307050127_0002 for user-log deletion with retainTimeStamp:1373472921775 >2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding >job_201307051500_0001 for user-log deletion with retainTimeStamp:1373472921775 >2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding >job_201307051634_0001 for user-log deletion with retainTimeStamp:1373472921775 >2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding >job_201307032034_0001 for user-log deletion with retainTimeStamp:1373472921775 >2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding >job_201307051203_0001 for user-log deletion with retainTimeStamp:1373472921775 >2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding >job_201307051611_0001 for user-log deletion with retainTimeStamp:1373472921775 >2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker: Failed to >get system directory... >2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed to >get system directory... >2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed to >get system directory... >2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed to >get system directory... >2013-07-09 12:15:33,738 INFO org.apache.hadoop.mapred.TaskTracker: Failed to >get system directory... >2013-07-09 12:15:35,802 INFO org.apache.hadoop.mapred.TaskTracker: Failed to >get system directory... >2013-07-09 12:15:37,308 INFO org.apache.hadoop.mapred.TaskTracker: Failed to >get system directory... >2013-07-09 12:15:41,620 INFO org.apache.hadoop.mapred.TaskTracker: Failed to >get system directory... >2013-07-09 12:15:44,310 INFO org.apache.hadoop.mapred.TaskTracker: Failed to >get system directory... >2013-07-09 12:15:48,939 INFO org.apache.hadoop.mapred.TaskTracker: Failed to >get system directory... >2013-07-09 12:15:50,121 INFO org.apache.hadoop.mapred.TaskTracker: Failed to >get system directory... >2013-07-09 12:15:52,549 INFO org.apache.hadoop.mapred.TaskTracker: Failed to >get system directory... >2013-07-09 12:15:53,751 INFO org.apache.hadoop.mapred.TaskTracker: Failed to >get system directory... >2013-07-09 12:15:56,464 INFO org.apache.hadoop.mapred.TaskTracker: Failed to >get system directory... >2013-07-09 12:15:58,423 INFO org.apache.hadoop.mapred.TaskTracker: Failed to >get system directory... >2013-07-09 12:16:03,413 INFO org.apache.hadoop.mapred.TaskTracker: Failed to >get system directory... >2013-07-09 12:16:04,567 INFO org.apache.hadoop.mapred.TaskTracker: Failed to >get system directory... >2013-07-09 12:16:08,177 INFO org.apache.hadoop.mapred.TaskTracker: Failed to >get system directory... > >" > > >Please help. > > > >Thanks and Regards, > >Siddharth > > > > > >
