[
https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504502#comment-13504502
]
Avner BenHanoch commented on MAPREDUCE-4049:
--------------------------------------------
_Alejandro,_
With all due respect, I think that something in your behavior is inappropriate:
* You were never involved in this issue; still you gave yourself the liberty
to make it a sub issue of your supported MAPREDUCE-2454 issue, without
consulting anyone.
* This is especially inappropriate since MAPREDUCE-2454 is disputable and has
its acceptance problems regardless of my issue. Hence, its acceptance problems
will affect my issue.
* Your justification *"As all this JIRAs are small, I think we'll be able to
move fast with all of them."* is inappropriate since you actually created a
linkage that will surely postpone my issue instead of leaving each issue to
progress at its own pace!
* It is not the first time that the persons behind MAPREDUCE-2454 try to
disturb this JIRA issue.
Apparently, I don't have the privileges to break this "sub task" linkage;
hence, I am asking that you or someone else will do it.
I am welcoming any comment coming from a professional place with the simple
target of making Hadoop better. Having that said, I feel that the way you
blitzed my patch with any possible patty comment, sometime with disputable
claims, just before the patch is about to be accepted – is unfair,
unprofessional and unfriendly. Especially considering your complete silence
since this JIRA issue has started.
I am not sure that commenting in a blitz way will increase the quality of
hadoop. For example:
{quote}
"Checking for shuffleConsumerPlugin != null before closing it seems redundant,
you would have never got there if shufflePlugin is NULL."
{quote}
This is your mistake (I'll reach there in case isLocal == true). *There is no
option to remove the nullity check!*
{quote}
"Visibility annotations for the ShuffleConsumerPlugin, ShuffleContext, should
be Unstable"
{quote}
I think it is inappropriate to declare plugin interface as "Unstable", since it
must stay stable for 3rd party vendors.
--- --- --- ---
Personally, I have no problem to implement all the rest of your comments. It
should be very easy for me. Still, I am raising few points for consideration
regarding your following comments:
{quote}
"The Shuffle class should be renamed to DefaultShuffle."
"The ShuffleConsumerPlugin should be renamed to Shuffle."
{quote}
I chose the term 'ShuffleConsumerPlugin' and not something like 'Shuffle',
because it clarifies that we are in a *plugin* of *ShuffleConsumer*, rather
than a *builtin* *ShuffleProvider/ShuffleHandler*. Also, I didn't take the
liberty to rename core classes of Hadoop.
{quote}
"ShuffleConsumerPlugin, getShuffleConsumerPlugin() method is not required,
instead use the ReflectionUtils directly in the ReducerTask class."
{quote}
Here, I only followed existing convention of Hadoop as shown in
ResourceCalculatorPlugin.getResourceCalculatorPlugin(). Personally, I'll be
glad to follow your advice, and even to go one step further and make
ShuffleConsumerPlugin an interface instead of AbstractClass.
{quote}
"use 'mapreduce.job.reduce.shuffle.class' to be consistent with MAPREDUCE-2454."
{quote}
Here I chose 'mapreduce.shuffle…', since I think it is consistent with the
current convention in hadoop-3 configuration.
--- --- --- ---
I can tell you that Arun & Todd didn't make it easy for me with their requests
from this patch so far. Still, I understand, respect and accept all their
comments. I am sure that everyone involved only want the best for Hadoop.
I suggest we hear Arun's consideration and move forward with the patch in the
best professional way.
_*Arun,*_
I think you are very familiar with both Hadoop/MapReduce and this JIRA issue
since its inception. You are also well familiar and involved with
MAPREDUCE-2454. It is also safe to say you know Alejandro and Asokan better
than you know me. I believe everyone involved will agree that your sole
interest is Hadoop's quality. *I am asking you and everyone else to help
progressing here.*
Avner
> plugin for generic shuffle service
> ----------------------------------
>
> Key: MAPREDUCE-4049
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: performance, task, tasktracker
> Affects Versions: 1.0.3, 1.1.0, 2.0.0-alpha, 3.0.0
> Reporter: Avner BenHanoch
> Labels: merge, plugin, rdma, shuffle
> Fix For: trunk
>
> Attachments: HADOOP-1.x.y.patch, Hadoop Shuffle Plugin Design.rtf,
> mapreduce-4049.patch, mapreduce-4049.patch, mapreduce-4049.patch,
> mapreduce-4049.patch
>
>
> Support generic shuffle service as set of two plugins: ShuffleProvider &
> ShuffleConsumer.
> This will satisfy the following needs:
> # Better shuffle and merge performance. For example: we are working on
> shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE,
> or Infiniband) instead of using the current HTTP shuffle. Based on the fast
> RDMA shuffle, the plugin can also utilize a suitable merge approach during
> the intermediate merges. Hence, getting much better performance.
> # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden
> dependency of NodeManager with a specific version of mapreduce shuffle
> (currently targeted to 0.24.0).
> References:
> # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu
> from Auburn University with others,
> [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf]
> # I am attaching 2 documents with suggested Top Level Design for both plugins
> (currently, based on 1.0 branch)
> # I am providing link for downloading UDA - Mellanox's open source plugin
> that implements generic shuffle service using RDMA and levitated merge.
> Note: At this phase, the code is in C++ through JNI and you should consider
> it as beta only. Still, it can serve anyone that wants to implement or
> contribute to levitated merge. (Please be advised that levitated merge is
> mostly suit in very fast networks) -
> [http://www.mellanox.com/content/pages.php?pg=products_dyn&product_family=144&menu_section=69]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
