Here is the problem I get when I run the Word Count sample with a file that is over 130MB. It looks like the map tasks never complete. From my usage I am assuming that there are 3 map tasks and 2 reduce tasks correct ? Also where can I see the intermediate files that are generated ?
Thanks Avinash java org.apache.hadoop.examples.WordCount -m 3 -r 2 /home/alakshman/hadoop-0.12.3/in-dir/test2.dat /tmp/out-dir 07/05/23 15:06:21 INFO mapred.InputFormatBase: Total input paths to process : 1 07/05/23 15:06:21 INFO mapred.JobClient: Running job: job_ltsbiw [EMAIL PROTECTED] examples]$ java org.apache.hadoop.examples.WordCount -m 2 -r 3 /home/alakshman/hadoop-0.12.3/in-dir/test2.dat /tmp/out-dir 07/05/23 15:06:30 INFO mapred.InputFormatBase: Total input paths to process : 1 07/05/23 15:06:30 INFO mapred.JobClient: Running job: job_if514q 07/05/23 15:06:31 INFO mapred.JobClient: map 0% reduce 0% 07/05/23 15:06:32 INFO mapred.LocalJobRunner: file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432 07/05/23 15:06:32 INFO mapred.JobClient: map 5% reduce 0% 07/05/23 15:06:33 INFO mapred.LocalJobRunner: file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432 07/05/23 15:06:33 INFO mapred.JobClient: map 13% reduce 0% 07/05/23 15:06:34 INFO mapred.LocalJobRunner: file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432 07/05/23 15:06:34 INFO mapred.JobClient: map 22% reduce 0% 07/05/23 15:06:35 INFO mapred.LocalJobRunner: file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432 07/05/23 15:06:35 INFO mapred.JobClient: map 30% reduce 0% 07/05/23 15:06:36 INFO mapred.LocalJobRunner: file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432 07/05/23 15:06:36 INFO mapred.JobClient: map 39% reduce 0% 07/05/23 15:06:37 INFO mapred.LocalJobRunner: file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432 07/05/23 15:06:37 INFO mapred.JobClient: map 46% reduce 0% 07/05/23 15:06:38 INFO mapred.LocalJobRunner: file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432 07/05/23 15:06:38 INFO mapred.JobClient: map 55% reduce 0% 07/05/23 15:06:39 INFO mapred.LocalJobRunner: file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432 07/05/23 15:06:39 INFO mapred.JobClient: map 63% reduce 0% 07/05/23 15:06:40 INFO mapred.LocalJobRunner: file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432 07/05/23 15:06:40 INFO mapred.JobClient: map 73% reduce 0% 07/05/23 15:06:41 INFO mapred.LocalJobRunner: file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432 07/05/23 15:06:41 INFO mapred.JobClient: map 81% reduce 0% 07/05/23 15:06:42 INFO mapred.LocalJobRunner: file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432 07/05/23 15:06:42 INFO mapred.JobClient: map 89% reduce 0% 07/05/23 15:06:43 INFO mapred.LocalJobRunner: file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432 07/05/23 15:06:43 INFO mapred.JobClient: map 92% reduce 0% 07/05/23 15:06:44 INFO mapred.LocalJobRunner: file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432 07/05/23 15:06:45 INFO mapred.LocalJobRunner: file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432 07/05/23 15:06:46 INFO mapred.LocalJobRunner: file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432 07/05/23 15:06:47 INFO mapred.LocalJobRunner: file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432 07/05/23 15:06:48 INFO mapred.LocalJobRunner: file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432 07/05/23 15:06:49 INFO mapred.LocalJobRunner: file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432 07/05/23 15:06:50 INFO mapred.LocalJobRunner: file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432 07/05/23 15:06:51 INFO mapred.LocalJobRunner: file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432 07/05/23 15:06:52 INFO mapred.LocalJobRunner: file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432 07/05/23 15:06:53 INFO mapred.LocalJobRunner: file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432 On 5/23/07 2:53 PM, "Arun C Murthy" <[EMAIL PROTECTED]> wrote: > On Wed, May 23, 2007 at 02:50:20PM -0700, Avinash Lakshman wrote: >> I want to setup Hadoop to run on a cluster of 3 nodes. Here is the setup of >> my hadoop-site.xml file : >> >> <configuration> >> <property> >> <name>fs.default.name</name> >> <value>hdfs:///</value> >> </property> > > Fix fs.default.name's value to <host>:<port> similar to your mapre.job.tracker > and give it a try... > > Arun > >> >> <property> >> <name>dfs.secondary.info.port</name> >> <value>10000</value> >> </property> >> >> <property> >> <name>mapred.job.tracker</name> >> <value>dev030.sctm.facebook.com:9001</value> >> </property> >> >> <property> >> <name>dfs.replication</name> >> <value>1</value> >> </property> >> >> </configuration> >> >> However whenever I start my DFS I get the following error : >> >> dev030.sctm.facebook.com: starting secondarynamenode, logging to >> /home/alakshman/hadoop-0.12.3/bin/../logs/hadoop-alakshman-secondarynamenode >> -dev030.sctm.facebook.com.out >> dev030.sctm.facebook.com: Exception in thread "main" >> java.lang.IllegalArgumentException: port out of range:-1 >> dev030.sctm.facebook.com: at >> java.net.InetSocketAddress.<init>(InetSocketAddress.java:118) >> dev030.sctm.facebook.com: at >> org.apache.hadoop.dfs.DataNode.createSocketAddr(DataNode.java:106) >> dev030.sctm.facebook.com: at >> org.apache.hadoop.dfs.SecondaryNameNode.<init>(SecondaryNameNode.java:93) >> dev030.sctm.facebook.com: at >> org.apache.hadoop.dfs.SecondaryNameNode.main(SecondaryNameNode.java:474) >> >> How do I fix this ? Please advice. >> >> Thanks >> A >>
