Wrong comparator used to merge files in Reduce phase
----------------------------------------------------
Key: HADOOP-1535
URL: https://issues.apache.org/jira/browse/HADOOP-1535
Project: Hadoop
Issue Type: Bug
Reporter: Vivek Ratan
As per the fix for HADOOP-485, we allow users to optionally provide a different
comparator to group values when calling the user's Reduce function. Devaraj and
I were looking at the code yesterday and we found that in ReduceTask.java, we
use the user-supplied comparator to merge the output files from the Map tasks
(we use the user-supplied comparator when creating a new SequenceFile.Sorter
object). This is incorrect as the comparator used to merge Map output files
should be the same as that used to create those files in the Map phase. The
user-supplied comparator for grouping values should be used only in the
iterator passed to the user's Reduce function (which is done correctly in the
code).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
