Owen, So far we did a couple of work-arounds and made the jobs to work. Now we are already downstream running other applications. When we (likely) hit the problem again, I will try to get heap profile.
Thanks -Christian On 12/8/08 9:10 AM, "Owen O'Malley" <[EMAIL PROTECTED]> wrote: > On Dec 8, 2008, at 8:02 AM, Christian Kunz wrote: > >> Comparing hadoop-default.xml of hadoop-0.18 with hadoop-0.17, didn't >> map.sort.class change from >> org.apache.hadoop.mapred.MergeSorter to >> org.apache.hadoop.util.QuickSort? > > Yes, but the quick sort is only used in the mapper. The reducer > already has sorted runs and therefore only needs an external merge sort. > > The primary change in 0.18 for the reducer was HADOOP-2095. What were > the values of io.sort.factor and io.file.buffer.size? > > Christian, can you get the heap profile for one of the reduces that is > failing? > > mapred.task.profile=true > mapred.task.profile.maps= <empty string, since we don't want any maps> > mapred.task.profile.reduces= <number of reduce to profile> > mapred.task.profile.params=- > agentlib:hprof=heap=sites,force=n,thread=y,verbose=n,file=%s > > -- Owen
