[
https://issues.apache.org/jira/browse/MAPREDUCE-4808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13550409#comment-13550409
]
Mariappan Asokan commented on MAPREDUCE-4808:
---------------------------------------------
Hi Arun,
Thanks for your comments. The design for the use cases like hash-join,
limit-N query, and so on is left to the creativity of the implementer of the
plugin. I did not want to mention any specific designs. For the case of
limit-N query, I created a test which contains one implementation of the plugin.
You mentioned RDMA shuffle as one of the alternative shuffle implementations.
The RDMA shuffle requires special hardware(infiniband card) which may not be
present in all Hadoop installations. RDMA based shuffle does not require
{{MergeManager}} because it is a combination of shuffle and merge implemented
in native code. There is no clear separation of shuffle and merge.
The current HTTP shuffle has been around in Hadoop for a long time and
functionally it will continue to work even with infiniband cards by using IP
over Infiniband(IPoIB) without requiring any native code.
I consider RDMA shuffle as a special case and it is not going to be very common
to warrant obsoleting the current separation of shuffle and merge. Besides, a
merge plugin does not preclude RDMA merge. A shuffle plugin can be used for
that purpose.
The {{MergeManager}} not only manages memory, it also coordinates with the
shuffle and manages mulitple merge passes. The interface I have defined
captures the methods needed for the above purposes. A single {{merge()}}
method will not suffice to take care of merging shuffled data.
-- Asokan
> Allow reduce-side merge to be pluggable
> ---------------------------------------
>
> Key: MAPREDUCE-4808
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4808
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Affects Versions: 2.0.2-alpha
> Reporter: Arun C Murthy
> Assignee: Mariappan Asokan
> Fix For: 2.0.3-alpha
>
> Attachments: COMBO-mapreduce-4809-4812-4808.patch,
> mapreduce-4808.patch, mapreduce-4808.patch, mapreduce-4808.patch,
> mapreduce-4808.patch, mapreduce-4808.patch, MergeManagerPlugin.pdf
>
>
> Allow reduce-side merge to be pluggable for MAPREDUCE-2454
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
