[ 
https://issues.apache.org/jira/browse/MAPREDUCE-434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron Kimball updated MAPREDUCE-434:
------------------------------------

    Attachment: MAPREDUCE-434.2.patch

Submitting new patch sync'd with trunk.

When testing this patch, I uncovered a problematic interaction between 
mapper/reducer tasks and the {{Merger}}. Specifically, {{MergeQueue}} uses a 
static field {{Merger.includeFinalMerge}} which is set to true in {{MapTask}}, 
but is never cleared.

Now that both task types can occur in the same JVM, and a {{Merger}} may be 
used in both (local reduce tasks did not actually instantiate a {{Merger}} in 
the old codepath), this flag must be made a per-{{MergeQueue}} setting. I 
removed the static field and pushed it into {{MergeQueue}} as a member field; 
the {{Merger.merge()}} methods used as constructors for {{MergeQueue}} now take 
a {{TaskType}} parameter which can be set to {{TaskType.MAP}} to achieve the 
same effect.

> local map-reduce job limited to single reducer
> ----------------------------------------------
>
>                 Key: MAPREDUCE-434
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-434
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>         Environment: local job tracker
>            Reporter: Yoram Arnon
>            Assignee: Aaron Kimball
>            Priority: Minor
>         Attachments: MAPREDUCE-434.2.patch, MAPREDUCE-434.patch
>
>
> when mapred.job.tracker is set to 'local', my setNumReduceTasks call is 
> ignored, and the number of reduce tasks is set at 1.
> This prevents me from locally debugging my partition function, which tries to 
> partition based on the number of reduce tasks.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply via email to