Re: problem scaling cluster up to 2 slaves

[Samuel LEMOINE]

Thanks for these precisions.
I've read the 1374 Jira page and it seems to be the same problem as 
mine, unless mine happends automaticaly and prevents me from running 
hadoop on 2 or more slaves.
Since it is still unsolved, and as I'm not some kind of hardcore 
coder/debugger, I think that I'll give up for the moment and keep 
running hadoop on a single slave. I have to go on the subject of my 
intership, which is not debugging hadoop, even if I had liked to find a 
way to run hadoop on a real cluster.
However, I stay tuned to answer any question on this issue.
About the virtual machines, my boss supplied me with 3 ubuntu servers 
hosted by a unique physical server (an AMD Opteron with 2GB Ram). Each 
virtual server is launched and managed by Xen 
(http://en.wikipedia.org/wiki/Xen)


Konstantin Shvachko a écrit :
Samuel LEMOINE wrote:

Thanks for your answer, you give me hope :)
TO answer your interrogations, I use the haddop script "start-all.sh" 
to launch hadoop, and each plateform is a JAVA 1.5.0 jdk running on 
ubuntu 7.0.4.

Thank you this was important.

Your few advices raise as many questions for me : you mention a 
discussion named "HADOOP-1374", what does it refer to ? I've crawled 
in the old mail archives with the keyword "1374" and found some 
messages about the tasktracker error report number 1374, I guess it's 
what you was talking about. 

This is a reference to a jira issue. Here is the link
http://issues.apache.org/jira/browse/HADOOP-1374

If it's the case, where should I see this error? (the tasktracker's 
logs don't seem to print any error)

Task trackers send all exceptions to the job tracker, which logs them 
in its log file.

Then, you tell me to try changing the order  in which the slaves 
start : I was using the start-all.sh so that I had no controle on 
that, but even with a different starting order (by the use of 
bin/hadoop datanode and bin/hadoop tasktracker on each slave), it 
doesn't solve the problem, still blocking during the reduce phase.

I tryed to kill tasktrackers, but I am not really able to determine 
which one I have to kill, actualy they both look the same, only the % 
changes. 

You can see the details of your jobs using web ui:
http://<jobtracker-hostname>:50030
under "*Running Jobs*" click on the job, then on the reduce. You can 
see there which task is assigned to which task tracker.
For failed tasks you can also see the exceptions that caused failures.
Without the exceptions it is hard to say what went wrong with the job.
BTW, the file system state can be seen too at
http://<namenode-hostname>:50070

I managed to finish a job by killing tasktrackers and restarting 
them, one after one, progressing step by step, but actually... it's 
like doing the job with only one slave, and it already worked before. 
There seems to get a serious problem with my reduce task when running 
on 2 tasktrackers (or more, I'm actually running on 3 tasktrackers 
now, as I thought it could unblock the scheme, but it didn't).

My last "more successful" attempt is not very encouraging, but I'll 
expose it in a more detailed way: I dispose of 4 plateforms (each one 
running with a 1.5.0 jdk under 7.0.4 ubuntu), I'll call them 90, 91, 
102 and 131 (their DHCP attributed ip actually). 131 is my coputer, 
the 3 other are 3 virtual machines.

What do you mean by a virtual machine?

131> bin/hadoop namenode -format
131> bin/hadoop namenode
131> bin/hadoop dfs -put /.../documents /user/hadoop/documents
90> bin/hadoop datanode
91> bin/hadoop datanode
102> bin/hadoop datanode
131> bin/hadoop jobtracker
90> bin/hadoop tasktracker
91> bin/hadoop tasktracker
102> bin/hadoop tasktracker
131> bin/hadoop --config conf/ jar hadoop-0.13.1-examples.jar grep 
/user/hadoop/documents /user/hadoop/results blabla

and here is what it gives:
07/08/02 09:20:04 INFO mapred.FileInputFormat: Total input paths to 
process : 7
07/08/02 09:20:04 INFO mapred.JobClient: Running job: job_0001
07/08/02 09:20:05 INFO mapred.JobClient:  map 0% reduce 0%
07/08/02 09:20:09 INFO mapred.JobClient:  map 25% reduce 0%
07/08/02 09:20:11 INFO mapred.JobClient:  map 50% reduce 0%
07/08/02 09:20:12 INFO mapred.JobClient:  map 75% reduce 0%
07/08/02 09:20:14 INFO mapred.JobClient:  map 100% reduce 0%
07/08/02 09:20:22 INFO mapred.JobClient:  map 100% reduce 11%
07/08/02 09:21:03 INFO mapred.JobClient:  map 100% reduce 4%
07/08/02 09:21:03 INFO mapred.JobClient: Task Id : 
task_0001_m_000001_0, Status : FAILED
07/08/02 09:21:03 WARN mapred.JobClient: Error reading task 
outputhadoop102.e-manation.com
07/08/02 09:21:03 WARN mapred.JobClient: Error reading task 
outputhadoop102.e-manation.com
07/08/02 09:21:03 INFO mapred.JobClient: Task Id : 
task_0001_m_000002_0, Status : FAILED
07/08/02 09:21:03 WARN mapred.JobClient: Error reading task 
outputhadoop102.e-manation.com
07/08/02 09:21:03 WARN mapred.JobClient: Error reading task 
outputhadoop102.e-manation.com
07/08/02 09:21:03 INFO mapred.JobClient: Task Id : 
task_0001_m_000003_0, Status : FAILED
07/08/02 09:21:03 WARN mapred.JobClient: Error reading task 
outputhadoop102.e-manation.com
07/08/02 09:21:03 WARN mapred.JobClient: Error reading task 
outputhadoop102.e-manation.com
07/08/02 09:21:03 INFO mapred.JobClient: Task Id : 
task_0001_m_000004_0, Status : FAILED
07/08/02 09:21:03 WARN mapred.JobClient: Error reading task 
outputhadoop102.e-manation.com
07/08/02 09:21:03 WARN mapred.JobClient: Error reading task 
outputhadoop102.e-manation.com
07/08/02 09:21:03 INFO mapred.JobClient: Task Id : 
task_0001_m_000005_0, Status : FAILED
07/08/02 09:21:03 WARN mapred.JobClient: Error reading task 
outputhadoop102.e-manation.com
07/08/02 09:21:03 WARN mapred.JobClient: Error reading task 
outputhadoop102.e-manation.com
07/08/02 09:21:03 INFO mapred.JobClient: Task Id : 
task_0001_r_000001_0, Status : FAILED
07/08/02 09:21:03 WARN mapred.JobClient: Error reading task 
outputhadoop102.e-manation.com
07/08/02 09:21:03 WARN mapred.JobClient: Error reading task 
outputhadoop102.e-manation.com
07/08/02 09:21:04 INFO mapred.JobClient:  map 87% reduce 4%
07/08/02 09:21:07 INFO mapred.JobClient:  map 100% reduce 4%
07/08/02 09:21:16 INFO mapred.JobClient:  map 100% reduce 11%
07/08/02 09:21:53 INFO mapred.JobClient:  map 100% reduce 6%
07/08/02 09:21:53 INFO mapred.JobClient: Task Id : 
task_0001_m_000001_1, Status : FAILED
07/08/02 09:21:53 WARN mapred.JobClient: Error reading task 
outputhadoop91.e-manation.com
07/08/02 09:21:53 WARN mapred.JobClient: Error reading task 
outputhadoop91.e-manation.com
07/08/02 09:21:53 INFO mapred.JobClient: Task Id : 
task_0001_m_000002_1, Status : FAILED
07/08/02 09:21:53 WARN mapred.JobClient: Error reading task 
outputhadoop91.e-manation.com
07/08/02 09:21:53 WARN mapred.JobClient: Error reading task 
outputhadoop91.e-manation.com
07/08/02 09:21:53 INFO mapred.JobClient: Task Id : 
task_0001_m_000004_1, Status : FAILED
07/08/02 09:21:53 WARN mapred.JobClient: Error reading task 
outputhadoop91.e-manation.com
07/08/02 09:21:53 WARN mapred.JobClient: Error reading task 
outputhadoop91.e-manation.com
07/08/02 09:21:53 INFO mapred.JobClient: Task Id : 
task_0001_r_000002_0, Status : FAILED
07/08/02 09:21:53 WARN mapred.JobClient: Error reading task 
outputhadoop91.e-manation.com
07/08/02 09:21:53 WARN mapred.JobClient: Error reading task 
outputhadoop91.e-manation.com
07/08/02 09:22:07 INFO mapred.JobClient:  map 100% reduce 33%
07/08/02 09:22:48 INFO mapred.JobClient: Task Id : 
task_0001_r_000002_1, Status : FAILED
07/08/02 09:22:48 WARN mapred.JobClient: Error reading task 
outputhadoop102.e-manation.com
07/08/02 09:22:48 WARN mapred.JobClient: Error reading task 
outputhadoop102.e-manation.com
07/08/02 09:22:58 INFO mapred.JobClient:  map 100% reduce 66%
07/08/02 09:23:17 INFO mapred.JobClient: Task Id : 
task_0001_r_000001_1, Status : FAILED
07/08/02 09:23:17 WARN mapred.JobClient: Error reading task 
outputhadoop91.e-manation.com
07/08/02 09:23:17 WARN mapred.JobClient: Error reading task 
outputhadoop91.e-manation.com
07/08/02 09:23:26 INFO mapred.JobClient:  map 100% reduce 100%
07/08/02 09:23:27 INFO mapred.JobClient: Job complete: job_0001
07/08/02 09:23:27 INFO mapred.JobClient: Counters: 9
07/08/02 09:23:27 INFO mapred.JobClient:   Map-Reduce Framework
07/08/02 09:23:27 INFO mapred.JobClient:     Map input records=12
07/08/02 09:23:27 INFO mapred.JobClient:     Map output records=2
07/08/02 09:23:27 INFO mapred.JobClient:     Map input bytes=84
07/08/02 09:23:27 INFO mapred.JobClient:     Map output bytes=30
07/08/02 09:23:27 INFO mapred.JobClient:     Combine input records=2
07/08/02 09:23:27 INFO mapred.JobClient:     Combine output records=2
07/08/02 09:23:27 INFO mapred.JobClient:     Reduce input groups=1
07/08/02 09:23:27 INFO mapred.JobClient:     Reduce input records=2
07/08/02 09:23:27 INFO mapred.JobClient:     Reduce output records=1
07/08/02 09:23:27 INFO mapred.FileInputFormat: Total input paths to 
process : 3
07/08/02 09:23:28 INFO mapred.JobClient: Running job: job_0002
07/08/02 09:23:29 INFO mapred.JobClient:  map 0% reduce 0%
07/08/02 09:23:37 INFO mapred.JobClient:  map 33% reduce 0%
07/08/02 09:23:38 INFO mapred.JobClient:  map 66% reduce 0%
07/08/02 09:23:39 INFO mapred.JobClient:  map 100% reduce 0%
07/08/02 09:23:47 INFO mapred.JobClient:  map 100% reduce 11%
07/08/02 09:24:04 INFO mapred.JobClient: Task Id : 
task_0002_r_000001_0, Status : FAILED
07/08/02 09:24:04 WARN mapred.JobClient: Error reading task 
outputhadoop102.e-manation.com
07/08/02 09:24:04 WARN mapred.JobClient: Error reading task 
outputhadoop102.e-manation.com
07/08/02 09:24:26 INFO mapred.JobClient:  map 100% reduce 7%
07/08/02 09:24:26 INFO mapred.JobClient: Task Id : 
task_0002_m_000001_0, Status : FAILED
07/08/02 09:24:26 WARN mapred.JobClient: Error reading task 
outputhadoop91.e-manation.com
07/08/02 09:24:26 WARN mapred.JobClient: Error reading task 
outputhadoop91.e-manation.com
07/08/02 09:24:26 INFO mapred.JobClient: Task Id : 
task_0002_r_000000_0, Status : FAILED
07/08/02 09:24:26 WARN mapred.JobClient: Error reading task 
outputhadoop91.e-manation.com
07/08/02 09:24:26 WARN mapred.JobClient: Error reading task 
outputhadoop91.e-manation.com
07/08/02 09:24:33 INFO mapred.JobClient:  map 100% reduce 33%
07/08/02 09:24:37 INFO mapred.JobClient:  map 100% reduce 66%
07/08/02 09:25:11 INFO mapred.JobClient: Task Id : 
task_0002_r_000001_1, Status : FAILED
07/08/02 09:25:11 WARN mapred.JobClient: Error reading task 
outputhadoop91.e-manation.com
07/08/02 09:25:11 WARN mapred.JobClient: Error reading task 
outputhadoop91.e-manation.com
07/08/02 09:25:17 INFO mapred.JobClient:  map 100% reduce 100%
07/08/02 09:25:18 INFO mapred.JobClient: Job complete: job_0002
07/08/02 09:25:18 INFO mapred.JobClient: Counters: 7
07/08/02 09:25:18 INFO mapred.JobClient:   Map-Reduce Framework
07/08/02 09:25:18 INFO mapred.JobClient:     Map input records=1
07/08/02 09:25:18 INFO mapred.JobClient:     Map output records=1
07/08/02 09:25:18 INFO mapred.JobClient:     Map input bytes=23
07/08/02 09:25:18 INFO mapred.JobClient:     Map output bytes=15
07/08/02 09:25:18 INFO mapred.JobClient:     Reduce input groups=1
07/08/02 09:25:18 INFO mapred.JobClient:     Reduce input records=1
07/08/02 09:25:18 INFO mapred.JobClient:     Reduce output records=1

Every time the WARN and FAIL appears is when I kill and restart a 
tasktracker.
What's really disheartening is that while the reduce job goes further 
and further, some tasktrackers become like "lazy", staying passive 
like unrecognized by the jobtracker or I don't know... so that the 
mess ends as if there were only 1 tasktracker slave working.
Actually, as soon as 2 tasktrackers or more are working at the same 
time, the job sticks...
Anyone, any idea ??

Sam





Konstantin Shvachko a écrit :

It seems like you are hitting the problem discussed in HADOOP-1374.
We still don't know what causes it. It would be very good if you 
share your experience in this issue.
Do you use hadoop scripts to run your tests? Which platform are you 
running them on?

In my experience sometimes it helps if you change the order in which 
the slaves start.
If that does not help try to kill the task-tracker that is stuck and 
restart it later.
The reduce task will be rescheduled to the other task-tracker and 
will succeed.

Thanks,
Konstantin

Samuel LEMOINE wrote:

Hi everyone !

I'm still trying to understand the way hadoop works, and the 
possibilities offered in parallelizing java applications with 
haddop (especially lucene-based ones).
For the moment, I've focused my efforts on the examples given (Grep 
and WordCount).
I've managed to make both of them work with a hadoop running as 
"file:///" namenode.
When I try it on dfs monde (namenode configured to my local ip), 
Grep still works, but WordCount doesn't anymore.
I manage to make Grep work with 1 computer for namenode and another 
for 1 datanode (obviously, WordCount still doesn't work).

But as soon as I try to built some kind of mini "real" cluster, 
with a namenode and n datanodes, with n=2, the map/reduce blocks. 
It doesn't produce any error, and the logs files don't show 
anything strange, the execution just freezes a little time after 
the beginning of the reduce task. I've noticed that the reduce task 
begins before the end of the map task when 2 slaves are availables, 
which is not the case with only 1 slave. I know that it's the 
expected behaviour, but I suspect it to be the cause of the freeze.
The execution is doesn't follow exactly the same way each time, 
sometimes it blocks on 91%map/31%reduce, other times 
89%map/15%reduce, but it's always at the beginning of the reduce 
task. I've tryed to let it run for a whole night in case it was 
just veeeeryyyy long, but it didn't go any further.
The 2 slaves are quite identical, and both of them work when alone 
as slave.

I'm blocked at this point for about 5 days, hanging around with no 
track to explore anymore. Any help will be greatly appreciated, any 
clue or so... if someone knows a good way for monitoring the 
distant java-tasks launched on the distant 
jobtrackers/tasktrackers, it's also welcome :)

Oh, a little precision that shouldn't change much things: the 2 
slaves are virtual machines, with 256MB of RAM each. All access to 
them are through ssh (which is quite boring when you change the 
config files a lot of times to try the different possibilities :D )


Samuel


PS: I promise to share my hadoop experience as soon as it takes a 
consistent shape. I anticipate rediging a tutorial for the company 
where i'm doing my internship, and then translate it in english for 
the wiki.
PS2: here follow the console messages for a few of my attempts;

console messages while running Grep on a 1master-2slaves dfs 
architecture:

/opt/java/bin/java -Didea.launcher.port=7540 
-Didea.launcher.bin.path=/opt/idea-6180/bin -Dfile.encoding=UTF-8 
-classpath 
/opt/jdk1.5.0_12/jre/lib/charsets.jar:/opt/jdk1.5.0_12/jre/lib/jce.jar:/opt/jdk1.5.0_12/jre/lib/jsse.jar:/opt/jdk1.5.0_12/jre/lib/plugin.jar:/opt/jdk1.5.0_12/jre/lib/deploy.jar:/opt/jdk1.5.0_12/jre/lib/javaws.jar:/opt/jdk1.5.0_12/jre/lib/rt.jar:/opt/jdk1.5.0_12/jre/lib/ext/localedata.jar:/opt/jdk1.5.0_12/jre/lib/ext/dnsns.jar:/opt/jdk1.5.0_12/jre/lib/ext/sunpkcs11.jar:/opt/jdk1.5.0_12/jre/lib/ext/sunjce_provider.jar:/home/samuel/IdeaProjects/hadoopTest/classes/test/hadoopTest:/home/samuel/IdeaProjects/hadoopTest/classes/production/hadoopTest:/home/samuel/commons-logging-1.1/commons-logging-1.1.jar:/home/samuel/IdeaProjects/hadoopTest/lib/log4j/log4j-1.2.14.jar:/home/samuel/IdeaProjects/hadoopTest/lib/commons-cli-1.1.jar:/home/samuel/IdeaProjects/hadoopTest/lib/http/commons-codec-1.3.jar:/home/samuel/IdeaProjects/hadoopTest/lib/http/commons-logging-1.0.4.jar:/home/samuel/IdeaProjects/hadoopTest/lib/http/httpclient-4.0-alpha1.jar:/home/samuel/IdeaProjects/hadoopTest/lib/http/ht 
tpcore-4.0-alpha5.jar:/home/samuel/IdeaProjects/hadoopTest/lib/commons-httpclient-3.0.1/commons-httpclient-3.0.1.jar:/home/samuel/IdeaProjects/hadoopTest/hadoop/conf:/home/samuel/hadoop-0.13.1/hadoop-0.13.1-core.jar:/opt/idea-6180/lib/idea_rt.jar 
com.intellij.rt.execution.application.AppMain 
com.lingway.hadoopScratchPad.Grep /user/hadoop/documents 
/user/hadoop/results blabla
07/08/01 11:14:01 INFO mapred.FileInputFormat: Total input paths to 
process : 19
07/08/01 11:14:03 INFO mapred.JobClient: Running job: job_0001
07/08/01 11:14:04 INFO mapred.JobClient:  map 0% reduce 0%
07/08/01 11:14:14 INFO mapred.JobClient:  map 10% reduce 0%
07/08/01 11:14:15 INFO mapred.JobClient:  map 21% reduce 0%
07/08/01 11:14:16 INFO mapred.JobClient:  map 26% reduce 0%
07/08/01 11:14:17 INFO mapred.JobClient:  map 31% reduce 0%
07/08/01 11:14:18 INFO mapred.JobClient:  map 42% reduce 0%
07/08/01 11:14:19 INFO mapred.JobClient:  map 47% reduce 0%
07/08/01 11:14:21 INFO mapred.JobClient:  map 57% reduce 0%
07/08/01 11:14:22 INFO mapred.JobClient:  map 63% reduce 0%
07/08/01 11:14:23 INFO mapred.JobClient:  map 73% reduce 0%
07/08/01 11:14:24 INFO mapred.JobClient:  map 78% reduce 0%
07/08/01 11:14:25 INFO mapred.JobClient:  map 84% reduce 0%
07/08/01 11:14:26 INFO mapred.JobClient:  map 89% reduce 0%
07/08/01 11:14:34 INFO mapred.JobClient:  map 89% reduce 15%

//and then doesn't go any further




******************************************************************************************** 


console messages while running WordCount on a single-node dfs 
architecture:

/opt/java/bin/java -Didea.launcher.port=7537 
-Didea.launcher.bin.path=/opt/idea-6180/bin -Dfile.encoding=UTF-8 
-classpath 
/opt/jdk1.5.0_12/jre/lib/charsets.jar:/opt/jdk1.5.0_12/jre/lib/jce.jar:/opt/jdk1.5.0_12/jre/lib/jsse.jar:/opt/jdk1.5.0_12/jre/lib/plugin.jar:/opt/jdk1.5.0_12/jre/lib/deploy.jar:/opt/jdk1.5.0_12/jre/lib/javaws.jar:/opt/jdk1.5.0_12/jre/lib/rt.jar:/opt/jdk1.5.0_12/jre/lib/ext/localedata.jar:/opt/jdk1.5.0_12/jre/lib/ext/dnsns.jar:/opt/jdk1.5.0_12/jre/lib/ext/sunpkcs11.jar:/opt/jdk1.5.0_12/jre/lib/ext/sunjce_provider.jar:/home/samuel/IdeaProjects/hadoopTest/classes/test/hadoopTest:/home/samuel/IdeaProjects/hadoopTest/classes/production/hadoopTest:/home/samuel/commons-logging-1.1/commons-logging-1.1.jar:/home/samuel/IdeaProjects/hadoopTest/lib/log4j/log4j-1.2.14.jar:/home/samuel/IdeaProjects/hadoopTest/lib/commons-cli-1.1.jar:/home/samuel/IdeaProjects/hadoopTest/lib/http/commons-codec-1.3.jar:/home/samuel/IdeaProjects/hadoopTest/lib/http/commons-logging-1.0.4.jar:/home/samuel/IdeaProjects/hadoopTest/lib/http/httpclient-4.0-alpha1.jar:/home/samuel/IdeaProjects/hadoopTest/lib/http/ht 
tpcore-4.0-alpha5.jar:/home/samuel/IdeaProjects/hadoopTest/lib/commons-httpclient-3.0.1/commons-httpclient-3.0.1.jar:/home/samuel/IdeaProjects/hadoopTest/hadoop/conf:/home/samuel/hadoop-0.13.1/hadoop-0.13.1-core.jar:/opt/idea-6180/lib/idea_rt.jar 
com.intellij.rt.execution.application.AppMain 
com.lingway.hadoopScratchPad.WordCount /user/hadoop/documents 
/user/hadoop/resultats
07/08/01 10:47:41 INFO mapred.FileInputFormat: Total input paths to 
process : 19
07/08/01 10:47:42 INFO mapred.JobClient: Running job: job_0004
07/08/01 10:47:43 INFO mapred.JobClient:  map 0% reduce 0%
07/08/01 10:47:48 INFO mapred.JobClient: Task Id : 
task_0004_m_000000_0, Status : FAILED
07/08/01 10:47:52 INFO mapred.JobClient: Task Id : 
task_0004_m_000001_0, Status : FAILED
07/08/01 10:47:57 INFO mapred.JobClient: Task Id : 
task_0004_m_000002_0, Status : FAILED
07/08/01 10:48:01 INFO mapred.JobClient: Task Id : 
task_0004_m_000003_0, Status : FAILED
07/08/01 10:48:05 INFO mapred.JobClient: Task Id : 
task_0004_m_000004_0, Status : FAILED
07/08/01 10:48:10 INFO mapred.JobClient: Task Id : 
task_0004_m_000005_0, Status : FAILED
07/08/01 10:48:14 INFO mapred.JobClient: Task Id : 
task_0004_m_000006_0, Status : FAILED
07/08/01 10:48:18 INFO mapred.JobClient: Task Id : 
task_0004_m_000007_0, Status : FAILED
07/08/01 10:48:24 INFO mapred.JobClient: Task Id : 
task_0004_m_000008_0, Status : FAILED
07/08/01 10:48:28 INFO mapred.JobClient: Task Id : 
task_0004_m_000009_0, Status : FAILED
07/08/01 10:48:32 INFO mapred.JobClient: Task Id : 
task_0004_m_000010_0, Status : FAILED
07/08/01 10:48:36 INFO mapred.JobClient: Task Id : 
task_0004_m_000011_0, Status : FAILED
07/08/01 10:48:41 INFO mapred.JobClient: Task Id : 
task_0004_m_000012_0, Status : FAILED
07/08/01 10:48:45 INFO mapred.JobClient: Task Id : 
task_0004_m_000013_0, Status : FAILED
07/08/01 10:48:49 INFO mapred.JobClient: Task Id : 
task_0004_m_000014_0, Status : FAILED
07/08/01 10:48:54 INFO mapred.JobClient: Task Id : 
task_0004_m_000015_0, Status : FAILED
07/08/01 10:48:58 INFO mapred.JobClient: Task Id : 
task_0004_m_000016_0, Status : FAILED
07/08/01 10:49:02 INFO mapred.JobClient: Task Id : 
task_0004_m_000017_0, Status : FAILED
07/08/01 10:49:06 INFO mapred.JobClient: Task Id : 
task_0004_m_000018_0, Status : FAILED
07/08/01 10:49:11 INFO mapred.JobClient: Task Id : 
task_0004_m_000000_1, Status : FAILED
07/08/01 10:49:15 INFO mapred.JobClient: Task Id : 
task_0004_m_000000_2, Status : FAILED
07/08/01 10:49:19 INFO mapred.JobClient:  map 100% reduce 100%
Exception in thread "main" java.io.IOException: Job failed!
   at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:604)
   at com.lingway.hadoopScratchPad.WordCount.main(WordCount.java:145)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 

   at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 

   at java.lang.reflect.Method.invoke(Method.java:585)
   at 
com.intellij.rt.execution.application.AppMain.main(AppMain.java:90)

Process finished with exit code 1








******************************************************************************************** 



console messages while running WordCount on a 1master-1slave dfs 
architecture:

/opt/java/bin/java -Didea.launcher.port=7539 
-Didea.launcher.bin.path=/opt/idea-6180/bin -Dfile.encoding=UTF-8 
-classpath 
/opt/jdk1.5.0_12/jre/lib/charsets.jar:/opt/jdk1.5.0_12/jre/lib/jce.jar:/opt/jdk1.5.0_12/jre/lib/jsse.jar:/opt/jdk1.5.0_12/jre/lib/plugin.jar:/opt/jdk1.5.0_12/jre/lib/deploy.jar:/opt/jdk1.5.0_12/jre/lib/javaws.jar:/opt/jdk1.5.0_12/jre/lib/rt.jar:/opt/jdk1.5.0_12/jre/lib/ext/localedata.jar:/opt/jdk1.5.0_12/jre/lib/ext/dnsns.jar:/opt/jdk1.5.0_12/jre/lib/ext/sunpkcs11.jar:/opt/jdk1.5.0_12/jre/lib/ext/sunjce_provider.jar:/home/samuel/IdeaProjects/hadoopTest/classes/test/hadoopTest:/home/samuel/IdeaProjects/hadoopTest/classes/production/hadoopTest:/home/samuel/commons-logging-1.1/commons-logging-1.1.jar:/home/samuel/IdeaProjects/hadoopTest/lib/log4j/log4j-1.2.14.jar:/home/samuel/IdeaProjects/hadoopTest/lib/commons-cli-1.1.jar:/home/samuel/IdeaProjects/hadoopTest/lib/http/commons-codec-1.3.jar:/home/samuel/IdeaProjects/hadoopTest/lib/http/commons-logging-1.0.4.jar:/home/samuel/IdeaProjects/hadoopTest/lib/http/httpclient-4.0-alpha1.jar:/home/samuel/IdeaProjects/hadoopTest/lib/http/ht 
tpcore-4.0-alpha5.jar:/home/samuel/IdeaProjects/hadoopTest/lib/commons-httpclient-3.0.1/commons-httpclient-3.0.1.jar:/home/samuel/IdeaProjects/hadoopTest/hadoop/conf:/home/samuel/hadoop-0.13.1/hadoop-0.13.1-core.jar:/opt/idea-6180/lib/idea_rt.jar 
com.intellij.rt.execution.application.AppMain 
com.lingway.hadoopScratchPad.WordCount /user/hadoop/documents 
/user/hadoop/resultats
07/08/01 11:08:24 INFO mapred.FileInputFormat: Total input paths to 
process : 19
07/08/01 11:08:26 INFO mapred.JobClient: Running job: job_0001
07/08/01 11:08:27 INFO mapred.JobClient:  map 0% reduce 0%
07/08/01 11:08:37 INFO mapred.JobClient: Task Id : 
task_0001_m_000002_0, Status : FAILED
07/08/01 11:08:37 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:08:37 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:08:37 INFO mapred.JobClient: Task Id : 
task_0001_m_000000_0, Status : FAILED
07/08/01 11:08:37 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:08:37 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:08:41 INFO mapred.JobClient: Task Id : 
task_0001_m_000005_0, Status : FAILED
07/08/01 11:08:41 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:08:41 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:08:42 INFO mapred.JobClient: Task Id : 
task_0001_m_000004_0, Status : FAILED
07/08/01 11:08:42 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:08:42 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:08:46 INFO mapred.JobClient: Task Id : 
task_0001_m_000006_0, Status : FAILED
07/08/01 11:08:46 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:08:46 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:08:47 INFO mapred.JobClient: Task Id : 
task_0001_m_000001_0, Status : FAILED
07/08/01 11:08:47 INFO mapred.JobClient: Task Id : 
task_0001_m_000007_0, Status : FAILED
07/08/01 11:08:47 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:08:48 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:08:48 INFO mapred.JobClient: Task Id : 
task_0001_m_000003_0, Status : FAILED
07/08/01 11:08:52 INFO mapred.JobClient: Task Id : 
task_0001_m_000008_0, Status : FAILED
07/08/01 11:08:52 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:08:52 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:08:52 INFO mapred.JobClient: Task Id : 
task_0001_m_000009_0, Status : FAILED
07/08/01 11:08:52 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:08:52 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:08:57 INFO mapred.JobClient: Task Id : 
task_0001_m_000010_0, Status : FAILED
07/08/01 11:08:57 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:08:57 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:08:57 INFO mapred.JobClient: Task Id : 
task_0001_r_000000_0, Status : FAILED
07/08/01 11:08:57 INFO mapred.JobClient: Task Id : 
task_0001_m_000011_0, Status : FAILED
07/08/01 11:08:57 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:08:57 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:09:02 INFO mapred.JobClient: Task Id : 
task_0001_m_000012_0, Status : FAILED
07/08/01 11:09:02 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:09:02 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:09:02 INFO mapred.JobClient: Task Id : 
task_0001_m_000013_0, Status : FAILED
07/08/01 11:09:02 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:09:02 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:09:02 INFO mapred.JobClient: Task Id : 
task_0001_m_000002_1, Status : FAILED
07/08/01 11:09:02 INFO mapred.JobClient: Task Id : 
task_0001_m_000000_1, Status : FAILED
07/08/01 11:09:07 INFO mapred.JobClient: Task Id : 
task_0001_m_000014_0, Status : FAILED
07/08/01 11:09:07 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:09:07 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:09:07 INFO mapred.JobClient: Task Id : 
task_0001_m_000015_0, Status : FAILED
07/08/01 11:09:07 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:09:07 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:09:11 INFO mapred.JobClient: Task Id : 
task_0001_m_000016_0, Status : FAILED
07/08/01 11:09:11 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:09:11 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:09:12 INFO mapred.JobClient: Task Id : 
task_0001_m_000017_0, Status : FAILED
07/08/01 11:09:12 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:09:12 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:09:16 INFO mapred.JobClient: Task Id : 
task_0001_m_000018_0, Status : FAILED
07/08/01 11:09:16 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:09:16 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:09:16 INFO mapred.JobClient: Task Id : 
task_0001_m_000001_1, Status : FAILED
07/08/01 11:09:16 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:09:16 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:09:17 INFO mapred.JobClient: Task Id : 
task_0001_m_000004_1, Status : FAILED
07/08/01 11:09:18 INFO mapred.JobClient: Task Id : 
task_0001_m_000005_1, Status : FAILED
07/08/01 11:09:21 INFO mapred.JobClient: Task Id : 
task_0001_m_000000_2, Status : FAILED
07/08/01 11:09:21 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:09:21 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:09:21 INFO mapred.JobClient: Task Id : 
task_0001_m_000003_1, Status : FAILED
07/08/01 11:09:21 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:09:21 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:09:27 INFO mapred.JobClient:  map 100% reduce 100%
07/08/01 11:09:27 INFO mapred.JobClient: Task Id : 
task_0001_m_000001_2, Status : FAILED
07/08/01 11:09:27 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
07/08/01 11:09:27 WARN mapred.JobClient: Error reading task 
outputubuntu704.e-manation.com
Exception in thread "main" java.io.IOException: Job failed!
   at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:604)
   at com.lingway.hadoopScratchPad.WordCount.main(WordCount.java:145)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 

   at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 

   at java.lang.reflect.Method.invoke(Method.java:585)
   at 
com.intellij.rt.execution.application.AppMain.main(AppMain.java:90)

Process finished with exit code 1