Here I comes, just another problem I encountered when I am running
hadoop in distribution mode:
The mapping function seems be running alright, as the number of nodes in
conf/slave increases, but the reduce function is still slow, and seems
never going to complete the job...(it's not stopping, it just really slow).
Is there anyway I can make the reduce process faster?
regards
yu-yang
yu-yang chen wrote:
Thanks to all those answered my question, I solved the problem finally:
After getting path not found error, I manually deleted the dfs file
system in
/tmp/hadoop-yyc04/mapred/system
and reformat( bin/hadoop namenode -format) again, which then worked.
Just in case someone else run into the same problem as me in the
future, here is one suggestion :)
ps: to shut down the dfs, it actually takes a while before it kills
all the namenodes completely...so after you run bin/stop-all.sh, you
might want to wait for few second before start again.
Thanks again
Regards
yu-yang
Hairong Kuang wrote:
Please make sure to shut down your dfs before you restart it. To
shut down
your dfs, you can either run stop-dfs.sh on namenode or go to each
datanode
kill each datanode java process.
After you start your dfs, you may check your cluster status by either
running "hadoop dfsadmin -report" or browsing to a web UI, which by
default
should be at http://namenode:50070.
Hairong
-----Original Message-----
From: yu-yang chen [mailto:[EMAIL PROTECTED] Sent: Thursday, May
24, 2007 4:14 AM
To: [email protected]
Subject: Re: Beginner to Hadoop, please help
Oh, here is the error reported in the log:
2007-05-23 23:34:01,716 WARN org.apache.hadoop.dfs.StateChange: DIR*
FSDirectory.unprotectedDelete: failed to remove
/tmp/hadoop-yyc04/mapred/system because it does not exist
2007-05-23 23:34:01,723 ERROR org.apache.hadoop.dfs.NameNode:
java.io.FileNotFoundException: Parent path does not exist:
/tmp/hadoop-yyc04/mapred/system
at
org.apache.hadoop.dfs.FSDirectory$INode.addNode(FSDirectory.java:195)
but i checked, and /tmp/hadoop-yyc04/mapred/system does exist in the
system....how should I fix this?
Thank you for answering
yuyang
yu-yang chen wrote:
Hello, thank you for the reply.
I checked the hosts list and all the name I am using is fully
resolvable. And I even tried to use ip address, but still obtain:
Retrying connect to server:
camelot01-ib.doc.ic.ac.uk/10.100.1.1:9000. Already tried 1 time(s)
....
Retrying connect to server:
camelot01-ib.doc.ic.ac.uk/10.100.1.1:9000. Already tried 10 time(s)
I read some post in internet, and thinking could it be my nadenode
problem? which port 9000 was not opened..
this is what happened when i do bin/start-*.sh
it shows: datanode running as process 32322. Stop it first.......is
this normal? why stop it first?
I checked the process list on the datanode, there are 3 java
processes running, but the port 9000 wasn't Listening...what is the
possible cause?
camelot01% bin/start-dfs.sh
starting namenode, logging to
/homes/yyc04/hadoop-install/hadoop/bin/../logs/hadoop-yyc04-namenode-c
amelot01.out
camelot01-ib.doc.ic.ac.uk: datanode running as process 32322. Stop
it first.
localhost: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
localhost: @ WARNING: POSSIBLE DNS SPOOFING DETECTED! @
localhost: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
localhost: The RSA host key for localhost has changed,
localhost: and the key for the according IP address 146.169.2.131
localhost: is unchanged. This could either mean that
localhost: DNS SPOOFING is happening or the IP address for the host
localhost: and its host key have changed at the same time.
localhost: Offending key for IP in /homes/yyc04/.ssh/known_hosts:7
localhost: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
localhost: @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
localhost: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
localhost: IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
localhost: Someone could be eavesdropping on you right now
(man-in-the-middle attack)!
localhost: It is also possible that the RSA host key has just been
changed.
localhost: The fingerprint for the RSA key sent by the remote host is
localhost: 25:2d:94:54:7f:7b:a6:1f:eb:26:83:8f:9b:f8:b0:a4.
localhost: Please contact your system administrator.
localhost: Add correct host key in /homes/yyc04/.ssh/known_hosts to
get rid of this message.
localhost: Offending key in /homes/yyc04/.ssh/known_hosts:24
localhost: Password authentication is disabled to avoid
man-in-the-middle attacks.
localhost: Keyboard-interactive authentication is disabled to avoid
man-in-the-middle attacks.
localhost: X11 forwarding is disabled to avoid man-in-the-middle
attacks.
localhost: starting secondarynamenode, logging to
/homes/yyc04/hadoop-install/hadoop/bin/../logs/hadoop-yyc04-secondaryn
amenode-camelot01.out
camelot01% bin/start-
start-all.sh* start-dfs.sh* start-mapred.sh*
camelot01% bin/start-mapred.sh
jobtracker running as process 32604. Stop it first.
camelot01-ib.doc.ic.ac.uk: tasktracker running as process 343. Stop
it first.
this is my hadoop-site.xml
property>
<name>fs.default.name</name>
<value>camelot-ib.doc.ic.ac.uk:9000</value>
</property>
<property>
<name>mapred.job.tracker</name>
<value>camelot-ib.doc.ic.ac.uk:9001</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
Thank you all for answering, and if possible, help me with this case
again~
yu-yang
Wayne Liu wrote:
2007/5/24, yu-yang chen <[EMAIL PROTECTED]>:
Hello, after change the domain name, it still fails.
I just realise something:
when i run :
camelot01% bin/hadoop jar hadoop-0.12.3-test.jar testrpc
Well, I think you'd better check the file /etc/hosts.
for example,my computer name : wayne , so the file hosts may be like
this:
127.0.0.1 localhost
202.197.18.27 wayne
If you are not sure about this,just have a try to ping your
computer name.
