[
https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13556524#comment-13556524
]
Alejandro Abdelnur commented on MAPREDUCE-4049:
-----------------------------------------------
Regarding 'new features in Hadoop-1'. Small or big, this is a new feature and
it should be treated as such. I'm all for having this in Hadoop 1. If you want
I can start the discussion in common-dev@.
Regarding "Again, throughout the lifetime of this JIRA issue,...", I see
different ways this can be done:
* Keep everything in the same JIRA (as it is now) and wait till the whole patch
is ready
* Break the JIRA in 2 subtasks, consumer and producer side
** Do it in branch-1 directly
** Do it in a dev branch (seems an overkill)
I'm OK with any approach, your call.
Regarding "default implementation already implements the producer API".
Ahh, missed that because the initialize method it is not used.
With some minor tweaks to your patch I think we could get things done in a
simple way:
* Add to the TT a 'public Server getHttpServer()' method
* In the TT constructor, where the MapOutputServlet is added to the HttpServer
'server', remove that line and discover, instantiate and initialize the
provider plugin.
* Don't make the MapOutputServlet to extends the provider interface.
* The default provider should be a class that simply adds the MapOutputServlet
to the server via the TT.getHttpServer() method.
* Remove the logic to instantiate a custom single provider plugin.
A provider multiplexor would be a very simple class, something along the
following lines:
{code}
public class MultiShuffleProviderPlugin implements ShuffleProviderPlugin {
public static final String PLUGIN_CLASSES =
"hadoop.mapreduce.multi.shuffle.provider.classes";
private ShuffleProviderPlugin[] plugins;
public void initialize(TaskTracker tt) {
Configuration conf = tt.getJobConf();
Class[] klasses = conf.getClasses(PLUGIN_CLASSES,
DefaultShuffleProvider.class);
//LOG INFO list of plugin classes
plugins = new ShuffleProviderPlugin[klasses.length];
for (int i = 0; i < klasses.length; i++) {
plugins[i] = ReflectionUtils.newInstance(klasses[i], conf);
}
for (ShuffleProviderPlugin plugin : plugins) {
plugin.initialize(tt);
}
}
public void destroy() {
if (plugins != null) {
for (ShuffleProviderPlugin plugin : plugins) {
try {
plugin.destroy();
} catch (Throwable ex) {
//LOG WARN and ignore exception
}
}
}
}
}
{code}
And the default provider class would be:
{code}
public static class DefaultShuffleProviderPlugin implements
ShuffleProviderPlugin {
public void initialize(TaskTracker tt) {
tt.getHttpServer().addInternalServlet("mapOutput", "/mapOutput",
MapOutputServlet.class);
}
public void destroy() {
}
}
{code}
> plugin for generic shuffle service
> ----------------------------------
>
> Key: MAPREDUCE-4049
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: performance, task, tasktracker
> Affects Versions: 1.0.3, 1.1.0, 2.0.0-alpha, 3.0.0
> Reporter: Avner BenHanoch
> Assignee: Avner BenHanoch
> Labels: merge, plugin, rdma, shuffle
> Fix For: 3.0.0
>
> Attachments: HADOOP-1.x.y.patch, Hadoop Shuffle Plugin Design.rtf,
> MAPREDUCE-4049--branch-1.patch, mapreduce-4049.patch
>
>
> Support generic shuffle service as set of two plugins: ShuffleProvider &
> ShuffleConsumer.
> This will satisfy the following needs:
> # Better shuffle and merge performance. For example: we are working on
> shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE,
> or Infiniband) instead of using the current HTTP shuffle. Based on the fast
> RDMA shuffle, the plugin can also utilize a suitable merge approach during
> the intermediate merges. Hence, getting much better performance.
> # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden
> dependency of NodeManager with a specific version of mapreduce shuffle
> (currently targeted to 0.24.0).
> References:
> # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu
> from Auburn University with others,
> [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf]
> # I am attaching 2 documents with suggested Top Level Design for both plugins
> (currently, based on 1.0 branch)
> # I am providing link for downloading UDA - Mellanox's open source plugin
> that implements generic shuffle service using RDMA and levitated merge.
> Note: At this phase, the code is in C++ through JNI and you should consider
> it as beta only. Still, it can serve anyone that wants to implement or
> contribute to levitated merge. (Please be advised that levitated merge is
> mostly suit in very fast networks) -
> [http://www.mellanox.com/content/pages.php?pg=products_dyn&product_family=144&menu_section=69]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
