[
https://issues.apache.org/jira/browse/MAPREDUCE-4808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mariappan Asokan updated MAPREDUCE-4808:
----------------------------------------
Attachment: mapreduce-4808.patch
Hi Chris and Alejandro,
Thanks to both of you for your comments and suggestions. I spent some time
reflecting on them. I totally agree with Chris that the RDMA/network merge
will be different from the way the merge is done by the current
{{MergeManager.}} In that case a shuffle plugin is the way to go.
When the current HTTP based {{Shuffle}} is used, it makes sense to define a
{{MergeManagerPlugin}} to work with that. So based on Alejandro's suggestion,
I updated the patch to make {{MergeManagerPlugin}} work only with {{Shuffle.}}
I removed the method {{mergeLocalFiles()}} from the interface as Chris
suggested. The {{Shuffle}} class would instantiate the {{MergeManagerPlugin}}
object in its {{init()}} method.
By the end of the day, Hadoop users will be offered more choices: a shuffle
plugin for RDMA/network merge when the cluster nodes have hardware support and
a merge plugin that works with the current shuffle when the cluster does not
have special network hardware.
Please provide any feedback on the updated patch.
Once again, thanks to both of you for the time you are spending on this Jira.
-- Asokan
> Allow reduce-side merge to be pluggable
> ---------------------------------------
>
> Key: MAPREDUCE-4808
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4808
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Affects Versions: 2.0.2-alpha
> Reporter: Arun C Murthy
> Assignee: Mariappan Asokan
> Fix For: 2.0.3-alpha
>
> Attachments: COMBO-mapreduce-4809-4812-4808.patch,
> mapreduce-4808.patch, mapreduce-4808.patch, mapreduce-4808.patch,
> mapreduce-4808.patch, mapreduce-4808.patch, mapreduce-4808.patch,
> MergeManagerPlugin.pdf
>
>
> Allow reduce-side merge to be pluggable for MAPREDUCE-2454
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
