On Dec 8, 2008, at 8:02 AM, Christian Kunz wrote:
Comparing hadoop-default.xml of hadoop-0.18 with hadoop-0.17, didn't
map.sort.class change from
org.apache.hadoop.mapred.MergeSorter to
org.apache.hadoop.util.QuickSort?
Yes, but the quick sort is only used in the mapper. The reducer
already has sorted runs and therefore only needs an external merge sort.
The primary change in 0.18 for the reducer was HADOOP-2095. What were
the values of io.sort.factor and io.file.buffer.size?
Christian, can you get the heap profile for one of the reduces that is
failing?
mapred.task.profile=true
mapred.task.profile.maps= <empty string, since we don't want any maps>
mapred.task.profile.reduces= <number of reduce to profile>
mapred.task.profile.params=-
agentlib:hprof=heap=sites,force=n,thread=y,verbose=n,file=%s
-- Owen
