[
https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504633#comment-13504633
]
Alejandro Abdelnur commented on MAPREDUCE-4049:
-----------------------------------------------
Hi Avner,
I respectfully disagree with your opinion that my behavior is inappropriate.
First of all, it is not my intention to slow you this JIRA down, but to make
sure it is consistent with the related work in MAPREDUCE-2454 (you can see that
in my comments). If that requires a couple of extra days, is is a small price
to pay.
As an Apache Hadoop developer is my responsibility to review and provide
feedback on work posted by other developers, my usual triggers are area of
knowledge, related work and area of interest.
This JIRA is tightly related to MAPREDUCE-2454, there is not dispute on that.
Thus it should stay as a subtask of it.
MAPREDUCE-2454 is not disputable, as it has been commented in it JIRA, it is
almost ready, it was matter of breaking it up and doing an fast interactive
review of its parts. As far as I can tell, this is already happening there.
Now going to your comments on my review:
* Yes the *shuffleConsumerPlugin != null*, you are right, I've noticed that
after I've posted my comments, so you can disregard that done.
* On the marking the ShuffleConsumerPlugin, ShuffleContext as *unstable*, it is
not appropriate, Hadoop wants to keep the right of modifying these APIs in the
future, if hte need arises. You can also see this, no only in MAPREDUCE-2454,
but in several places where Hadoop provides pluggability (ie
ResourceManagement, authentication).
* On making the ShuffleConsumerPlugin and interface, that is a good idea, it
will align things with the other sub-tasks.
Looking forward to see the updated patch.
Cheers.
> plugin for generic shuffle service
> ----------------------------------
>
> Key: MAPREDUCE-4049
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: performance, task, tasktracker
> Affects Versions: 1.0.3, 1.1.0, 2.0.0-alpha, 3.0.0
> Reporter: Avner BenHanoch
> Labels: merge, plugin, rdma, shuffle
> Fix For: trunk
>
> Attachments: HADOOP-1.x.y.patch, Hadoop Shuffle Plugin Design.rtf,
> mapreduce-4049.patch, mapreduce-4049.patch, mapreduce-4049.patch,
> mapreduce-4049.patch
>
>
> Support generic shuffle service as set of two plugins: ShuffleProvider &
> ShuffleConsumer.
> This will satisfy the following needs:
> # Better shuffle and merge performance. For example: we are working on
> shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE,
> or Infiniband) instead of using the current HTTP shuffle. Based on the fast
> RDMA shuffle, the plugin can also utilize a suitable merge approach during
> the intermediate merges. Hence, getting much better performance.
> # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden
> dependency of NodeManager with a specific version of mapreduce shuffle
> (currently targeted to 0.24.0).
> References:
> # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu
> from Auburn University with others,
> [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf]
> # I am attaching 2 documents with suggested Top Level Design for both plugins
> (currently, based on 1.0 branch)
> # I am providing link for downloading UDA - Mellanox's open source plugin
> that implements generic shuffle service using RDMA and levitated merge.
> Note: At this phase, the code is in C++ through JNI and you should consider
> it as beta only. Still, it can serve anyone that wants to implement or
> contribute to levitated merge. (Please be advised that levitated merge is
> mostly suit in very fast networks) -
> [http://www.mellanox.com/content/pages.php?pg=products_dyn&product_family=144&menu_section=69]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
