[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13526802#comment-13526802
 ] 

Mariappan Asokan commented on MAPREDUCE-4812:
---------------------------------------------

Hi Arun,
  Sorry I did not get back sooner.  The intention of {{ReduceInputMerger}} 
interface is to have a pluggable {{MergeManager}} implementation.  For a 
non-local job, {{Shuffle}} and {{MergeManager}} interact and synchronize with 
each other using the three methods {{waitForInMemoryMerge(),}} {{reserve(),}} 
and {{close()}}.  So in order to use the {{Shuffle}} these methods are captured 
in {{ReduceInputMerger}} interface.  I renamed {{waitForInMemoryMerge()}} to a 
generic name {{waitForResource()}} since the plugin implementation may not have 
the concept of in-memory merge.
Since the return value from {{reserve()}} is {{MapOutput}}, I did some 
refactoring of {{MapOutput}} so that plugin can return its own implementation 
of it.  I kept the refactoring done on {{MapOutput}} in MAPREDUCE-4808.  With 
just MAPREDUCE-4812, an external plugin is not possible, but it has the core 
part of the concepts so that it is easy to review just {{ReduceInputMerger}} 
design.  Similarly, for a local job the input is coming from local files.  I 
enhanced {{ReduceInputMerger}} with one more method for this.  It is also kept 
in MAPREDUCE-4808.

Hope I explained well.  Please let me know if you have any more questions.

Thanks.

-- Asokan

                
> Create reduce input merger plugin in ReduceTask.java and pass it to Shuffle
> ---------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-4812
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4812
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>    Affects Versions: 2.0.2-alpha
>            Reporter: Mariappan Asokan
>            Assignee: Mariappan Asokan
>             Fix For: 2.0.3-alpha
>
>         Attachments: COMBO-mapreduce-4809-4812.patch, 
> COMBO-mapreduce-4809-4812.patch, mapreduce-4812.patch, mapreduce-4812.patch, 
> mapreduce-4812.patch, mapreduce-4812.patch, mapreduce-4812.patch
>
>
> This is part of MAPREDUCE-2454.  This further breaks down MAPREDUCE-4808

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Reply via email to