I'm not sure what to do about your specific problem, but here's how
to look inside DFS: use the hadoop executable in bin/ to execute
common filesystem operations (ls, cp, rm, &c). For instance:
bin/hadoop dfs -ls /
does an "ls /" on DFS.
Go to http://wiki.apache.org/lucene-hadoop/hadoop-0.1-dev/bin/
hadoop_dfs?highlight=%28dfs%29 for more detailed documentation. I
found it pretty intuitive, but watch out for the one important quirk:
"a path may be relative or absolute. An absolute path starts with a
'/', a relative path does not, and always relates to /user/
<currentUser>. There is no notion of current working directory."
~ Esteban
On May 23, 2007, at 3:12 PM, Avinash Lakshman wrote:
Here is the problem I get when I run the Word Count sample with a
file that
is over 130MB. It looks like the map tasks never complete. From my
usage I
am assuming that there are 3 map tasks and 2 reduce tasks correct ?
Also
where can I see the intermediate files that are generated ?
Thanks
Avinash
java org.apache.hadoop.examples.WordCount -m 3 -r 2
/home/alakshman/hadoop-0.12.3/in-dir/test2.dat /tmp/out-dir
07/05/23 15:06:21 INFO mapred.InputFormatBase: Total input paths to
process
: 1
07/05/23 15:06:21 INFO mapred.JobClient: Running job: job_ltsbiw
[EMAIL PROTECTED] examples]$ java
org.apache.hadoop.examples.WordCount -m 2
-r 3 /home/alakshman/hadoop-0.12.3/in-dir/test2.dat /tmp/out-dir
07/05/23 15:06:30 INFO mapred.InputFormatBase: Total input paths to
process
: 1
07/05/23 15:06:30 INFO mapred.JobClient: Running job: job_if514q
07/05/23 15:06:31 INFO mapred.JobClient: map 0% reduce 0%
07/05/23 15:06:32 INFO mapred.LocalJobRunner:
file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432
07/05/23 15:06:32 INFO mapred.JobClient: map 5% reduce 0%
07/05/23 15:06:33 INFO mapred.LocalJobRunner:
file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432
07/05/23 15:06:33 INFO mapred.JobClient: map 13% reduce 0%
07/05/23 15:06:34 INFO mapred.LocalJobRunner:
file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432
07/05/23 15:06:34 INFO mapred.JobClient: map 22% reduce 0%
07/05/23 15:06:35 INFO mapred.LocalJobRunner:
file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432
07/05/23 15:06:35 INFO mapred.JobClient: map 30% reduce 0%
07/05/23 15:06:36 INFO mapred.LocalJobRunner:
file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432
07/05/23 15:06:36 INFO mapred.JobClient: map 39% reduce 0%
07/05/23 15:06:37 INFO mapred.LocalJobRunner:
file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432
07/05/23 15:06:37 INFO mapred.JobClient: map 46% reduce 0%
07/05/23 15:06:38 INFO mapred.LocalJobRunner:
file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432
07/05/23 15:06:38 INFO mapred.JobClient: map 55% reduce 0%
07/05/23 15:06:39 INFO mapred.LocalJobRunner:
file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432
07/05/23 15:06:39 INFO mapred.JobClient: map 63% reduce 0%
07/05/23 15:06:40 INFO mapred.LocalJobRunner:
file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432
07/05/23 15:06:40 INFO mapred.JobClient: map 73% reduce 0%
07/05/23 15:06:41 INFO mapred.LocalJobRunner:
file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432
07/05/23 15:06:41 INFO mapred.JobClient: map 81% reduce 0%
07/05/23 15:06:42 INFO mapred.LocalJobRunner:
file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432
07/05/23 15:06:42 INFO mapred.JobClient: map 89% reduce 0%
07/05/23 15:06:43 INFO mapred.LocalJobRunner:
file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432
07/05/23 15:06:43 INFO mapred.JobClient: map 92% reduce 0%
07/05/23 15:06:44 INFO mapred.LocalJobRunner:
file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432
07/05/23 15:06:45 INFO mapred.LocalJobRunner:
file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432
07/05/23 15:06:46 INFO mapred.LocalJobRunner:
file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432
07/05/23 15:06:47 INFO mapred.LocalJobRunner:
file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432
07/05/23 15:06:48 INFO mapred.LocalJobRunner:
file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432
07/05/23 15:06:49 INFO mapred.LocalJobRunner:
file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432
07/05/23 15:06:50 INFO mapred.LocalJobRunner:
file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432
07/05/23 15:06:51 INFO mapred.LocalJobRunner:
file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432
07/05/23 15:06:52 INFO mapred.LocalJobRunner:
file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432
07/05/23 15:06:53 INFO mapred.LocalJobRunner:
file:/home/alakshman/hadoop-0.12.3/in-dir/test2.dat:0+33554432
On 5/23/07 2:53 PM, "Arun C Murthy" <[EMAIL PROTECTED]> wrote:
On Wed, May 23, 2007 at 02:50:20PM -0700, Avinash Lakshman wrote:
I want to setup Hadoop to run on a cluster of 3 nodes. Here is
the setup of
my hadoop-site.xml file :
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs:///</value>
</property>
Fix fs.default.name's value to <host>:<port> similar to your
mapre.job.tracker
and give it a try...
Arun
<property>
<name>dfs.secondary.info.port</name>
<value>10000</value>
</property>
<property>
<name>mapred.job.tracker</name>
<value>dev030.sctm.facebook.com:9001</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
However whenever I start my DFS I get the following error :
dev030.sctm.facebook.com: starting secondarynamenode, logging to
/home/alakshman/hadoop-0.12.3/bin/../logs/hadoop-alakshman-
secondarynamenode
-dev030.sctm.facebook.com.out
dev030.sctm.facebook.com: Exception in thread "main"
java.lang.IllegalArgumentException: port out of range:-1
dev030.sctm.facebook.com: at
java.net.InetSocketAddress.<init>(InetSocketAddress.java:118)
dev030.sctm.facebook.com: at
org.apache.hadoop.dfs.DataNode.createSocketAddr(DataNode.java:106)
dev030.sctm.facebook.com: at
org.apache.hadoop.dfs.SecondaryNameNode.<init>
(SecondaryNameNode.java:93)
dev030.sctm.facebook.com: at
org.apache.hadoop.dfs.SecondaryNameNode.main
(SecondaryNameNode.java:474)
How do I fix this ? Please advice.
Thanks
A
