Thanks much!!! Very fair answers.  (At least now I know it is not me
doing something wrong and it works now).  Now if only someone would add
a line to comment out/on in hadoop like so at the end....or do I have to
put that on the dev list?

 

#uncomment to debug processes

#export HADOOP_OPTS="-Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=9009
$HADOOP_OPTS"

 

At least that is how I was debugging these things and stepping through
hadoop code.

 

Thanks,

Dean

 

From: Aaron Eng [mailto:a...@maprtech.com] 
Sent: Wednesday, December 08, 2010 4:31 PM
To: hdfs-user@hadoop.apache.org
Subject: Re: wordcount example using local file system instead of
distributed one?

 

>Why did that work out of curiosity?
Each of the config files can be used to set only specific configuration
values.  So even though you had a valid config property/value, it didn't
apply because it wasn't in the right file.  So, why did it work? Because
thats the way it is coded.  A better question, why didn't it work with
your original config files? Because of poor usability.  I think you can
find a link to the config values that work in the various config files
somewhere on the Apache Hadoop site (that was vague, wasn't it?).

>Am I doing something wrong
Yes, you need to run start-dfs.sh on the node that you want to become
the namenode and you need to run start-mapred.sh on the node that you
want to become jobtracker.  Again, the reason is very poor usability and
bad scripting.  Of course, someone will inevitably say that you could
just right your own scripts to control the services...



On Wed, Dec 8, 2010 at 3:09 PM, Hiller, Dean (Contractor)
<dean.hil...@broadridge.com> wrote:

Sweeeeeeet!!!!! That worked...didn't see that in the docs at all.  Why
did that work out of curiosity?

 

Also, I run ./start-dfs.sh on node 1 and ./start-mapred.sh on node 2.
Am I doing something wrong in that

I cannot run those on any other nodes L.  Ie. If I run it on a different
node, the NameNode process does not

Run on node 1 and the JobTracker does not run on node 2.  It is like
those are locked to those boxes because

Of the config where the slaves file allow all the slaves to be started
just fine.  Am I doing something wrong or

Is that just simply always the case?

 

My config is mapred is set to jobtracker of node 2, and my masters only
has node 1 and my slaves has node 1 

And node 2.  Is there no mapred masters like file for when I run
./start-mapred?

 

I would think ./start-dfs.sh should work from any node since masters
file and slaves file contain all the nodes they

Need to start things on, but it doesn't seem to work?  (just want it to
be more seamless in case I am on the wrong node

as right now, it seems to only shut down some things and not all if done
from wrong node).

 

Thanks,

Dean

 

From: Aaron Eng [mailto:a...@maprtech.com] 
Sent: Wednesday, December 08, 2010 3:57 PM
To: hdfs-user@hadoop.apache.org
Subject: Re: wordcount example using local file system instead of
distributed one?

 

You will also need to restart services after that, in case that wasn't
obvious.

On Wed, Dec 8, 2010 at 2:56 PM, Aaron Eng <a...@maprtech.com> wrote:

Hi Dean,

Try removing the fs.default.name parameter from hdfs-site.xml and put it
in core-site.xml

 

On Wed, Dec 8, 2010 at 2:46 PM, Hiller, Dean (Contractor)
<dean.hil...@broadridge.com> wrote:

I run the following wordcount example(my hadoop shell seems to always
hit the local file system first so I had to add the hdfs...is that
normal??...I mean, I see it printing configDir= which is where I moved
the config dir and what I set the env var too which has the location in
the config files there but it still hits the local).

 

[r...@localhost hadoop]# ./bin/hadoop jar hadoop-0.20.2-examples.jar
wordcount

hdfs://206.88.43.8:54310/wordcount
hdfs://206.88.43.168:54310/wordcount-out

 

configDir=/mnt/mucho/hadoop-config/

classpath=/opt/hbase-install/hbase/hbase-0.20.6.jar:/opt/hbase-install/h
base/hba

se-0.20.6-test.jar:/mnt/mucho/hbase-config/:/opt/hbase-install/hbase/lib
/zookeep

er-3.2.2.jar

10/12/08 08:42:33 INFO input.FileInputFormat: Total input paths to
process : 13

org.apache.hadoop.ipc.RemoteException: java.io.FileNotFoundException:
File file:

/tmp/hadoop-root/mapred/system/job_201012080654_0010/job.xml does not
exist.

        at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSys

tem.java:361)

        at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.

java:245)

        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:192)

        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:142)

        at
org.apache.hadoop.fs.LocalFileSystem.copyToLocalFile(LocalFileSystem.

java:61)

        at
org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1197)

 

        at
org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:257)

 

        at
org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:234)

 

        at
org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:2993)

 

In case it helps, here is my hdfs-site.xml that is used by the daemons
started AND the client(is that an issue...using the same one)...

<configuration>

<property>

  <name>fs.default.name</name>

  <value>hdfs://206.88.43.168:54310</value>

</property>

<property>

  <name>hadoop.tmp.dir</name>

  <value>/opt/data/hadooptmp</value>

</property>

<property>

  <name>dfs.data.dir</name>

  <value>/opt/data/hadoop</value>

</property>

<property>

  <name>dfs.replication</name>

  <value>2</value>

</property>

</configuration>

 

 

This message and any attachments are intended only for the use of the
addressee and
may contain information that is privileged and confidential. If the
reader of the 
message is not the intended recipient or an authorized representative of
the
intended recipient, you are hereby notified that any dissemination of
this
communication is strictly prohibited. If you have received this
communication in
error, please notify us immediately by e-mail and delete the message and
any
attachments from your system.
 

 

 

This message and any attachments are intended only for the use of the
addressee and
may contain information that is privileged and confidential. If the
reader of the 
message is not the intended recipient or an authorized representative of
the
intended recipient, you are hereby notified that any dissemination of
this
communication is strictly prohibited. If you have received this
communication in
error, please notify us immediately by e-mail and delete the message and
any
attachments from your system.
 

 


This message and any attachments are intended only for the use of the addressee 
and
may contain information that is privileged and confidential. If the reader of 
the 
message is not the intended recipient or an authorized representative of the
intended recipient, you are hereby notified that any dissemination of this
communication is strictly prohibited. If you have received this communication in
error, please notify us immediately by e-mail and delete the message and any
attachments from your system.

Reply via email to