Steve, If you're going to use NFS, make sure your "hadoop.tmp.dir" property points to the mount point that is NFS. Can you change that property and restart the cluster and retry?
Regarding the HDFS issue, its hard to tell without logs. Did you see two nodes alive in the Web UI after configuring HDFS for two nodes and configuring MR to use HDFS? On Mon, Jul 23, 2012 at 11:23 PM, Steve Sonnenberg <steveis...@gmail.com> wrote: > Thanks Harsh, > > 1) I was using NFS > 2) I don't believe that anything under /tmp is distributed even when running > 3) When I use HDFS, it doesn't attempt to send ANY jobs to my second node > > Any clues? > > -steve > > > On Fri, Jul 20, 2012 at 11:52 PM, Harsh J <ha...@cloudera.com> wrote: >> >> A 2-node cluster is a fully-distributed cluster and cannot use a >> file:/// FileSystem as thats not a distributed filesystem (unless its >> an NFS mount). This explains why some of your tasks aren't able to >> locate an earlier written file on the /tmp dir thats probably >> available on the JT node alone, not the TT nodes. >> >> Use hdfs:// FS for fully-distributed operation. >> >> On Fri, Jul 20, 2012 at 10:06 PM, Steve Sonnenberg <steveis...@gmail.com> >> wrote: >> > I have a 2-node Fedora system and in cluster mode, I have the following >> > issue that I can't resolve. >> > >> > Hadoop 1.0.3 >> > I'm running with filesystem, file:/// and invoking the simple 'grep' >> > example >> > >> > hadoop jar hadoop-examples-1.0.3.jar grep inputdir outputdir >> > simple-pattern >> > >> > The initiator displays >> > >> > Error initializing attempt_201207201103_0003_m_000004_0: >> > java.io.FileNotFoundException: File >> > file:/tmp/hadoop-hadoop/mapred/system/job_201207201103_0003/jobToken >> > does >> > not exist. >> > getFileStatus(RawLocalFileSystem.java) >> > localizeJobTokenFile(TaskTracker.java:4268) >> > initializeJob(TaskTracker.java:1177) >> > localizeJob >> > run >> > >> > The /tmp/hadoop-hadoop/mapred/system directory only contains a >> > 'jobtracker.info' file (on all systems) >> > >> > On the target system, in the tasktracker log file, I get the following: >> > >> > 2012-07-20 11:35:59,954 DEBUG org.apache.hadoop.mapred.TaskTracker: Got >> > heartbeatResponse from JobTracker with responseId: 641 and 1 actions >> > 2012-07-20 11:35:59,954 INFO org.apache.hadoop.mapred.TaskTracker: >> > LaunchTaskAction (registerTask): attempt_201207201103_0003_m_000006_0 >> > task's >> > state:UNASSIGNED >> > 2012-07-20 11:35:59,954 INFO org.apache.hadoop.mapred.TaskTracker: >> > Trying to >> > launch : attempt_201207201103_0003_m_000006_0 which needs 1 slots >> > 2012-07-20 11:35:59,954 INFO org.apache.hadoop.mapred.TaskTracker: In >> > TaskLauncher, current free slots : 2 and trying to launch >> > attempt_201207201103_0003_m_000006_0 which needs 1 slots >> > 2012-07-20 11:35:59,955 WARN org.apache.hadoop.mapred.TaskTracker: Error >> > initializing attempt_201207201103_0003_m_000006_0: >> > java.io.FileNotFoundException: File >> > file:/tmp/hadoop-hadoop/mapred/system/job_201207201103_0003/jobToken >> > does >> > not exist. >> > at >> > >> > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:397) >> > at >> > >> > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) >> > at >> > >> > org.apache.hadoop.mapred.TaskTracker.localizeJobTokenFile(TaskTracker.java:4268) >> > at >> > >> > org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1177) >> > at >> > org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1118) >> > at >> > org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2430) >> > at java.lang.Thread.run(Thread.java:636) >> > >> > 2012-07-20 11:35:59,955 ERROR org.apache.hadoop.mapred.TaskStatus: >> > Trying to >> > set finish time for task attempt_201207201103_0003_m_000006_0 when no >> > start >> > time is set, stackTrace is : java.lang.Exception >> > at >> > org.apache.hadoop.mapred.TaskStatus.setFinishTime(TaskStatus.java:145) >> > at >> > >> > org.apache.hadoop.mapred.TaskTracker$TaskInProgress.kill(TaskTracker.java:3142) >> > at >> > org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2440) >> > at java.lang.Thread.run(Thread.java:636) >> > >> > On both systems, ownership of all files directories under >> > /tmp/hadoop-hadoop >> > is the user/group hadoop/hadoop. >> > >> > >> > Any ideas? >> > >> > Thanks >> > >> > >> > -- >> > Steve Sonnenberg >> > >> >> >> >> -- >> Harsh J > > > > > -- > Steve Sonnenberg > -- Harsh J
