Hi,
My cluster contains 22 DataNodes and Task Tracker each with 8 mapper slots and
4 reduce slots, each with 1.5G max heap size.
I use cloudera CDH 2
I have a specific job that is constantly failing in the reduce phase. I use 64
reducers and 64M block size and compress map output with LZO.
The same Exception which appears on all failed reduce tasks is :
The reduce copier failed
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.io.IOException: Intermediate merge failed
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2651)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(ReduceTask.java:2576)
Caused by: java.lang.RuntimeException: java.io.EOFException
at
org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:128)
at org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:373)
at org.apache.hadoop.util.PriorityQueue.downHeap(PriorityQueue.java:144)
at
org.apache.hadoop.util.PriorityQueue.adjustTop(PriorityQueue.java:103)
at
org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Merger.java:335)
at org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:350)
at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:156)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2635)
... 1 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:180)
at org.apache.hadoop.io.Text.readFields(Text.java:265)
at
com.conduit.UserLoginLog.Distinct2ActiveUsersKey.readFields(Distinct2ActiveUsersKey.java:114)
at
org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:122)
... 8 more
I do expect a relatively large map output from this job.
My mapred-site.xml contains
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>8</value>
<final>false</final>
</property>
<property>
<name>mapred.tasktracker.reduce.tasks.maximum</name>
<value>4</value>
<final>false</final>
</property>
<property>
<name>mapred.reduce.tasks</name>
<value>64</value>
</property>
<property>
<name>mapred.job.reduce.input.buffer.percent</name>
<value>0.9</value>
</property>
<property>
<name>mapred.job.shuffle.merge.percent</name>
<value>0.8</value>
</property>
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx1536m -Djava.library.path=/usr/lib/hadoop/lib/native
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false</value>
</property>
<property>
<name>mapreduce.map.output.compress</name>
<value>true</value>
</property>
<property>
<name>mapreduce.map.output.compress.codec</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
Can anyone speculate as to whats causing this? how can I at least make the job
complete ?
