[
https://issues.apache.org/jira/browse/MAPREDUCE-3060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13109872#comment-13109872
]
Milind Bhandarkar commented on MAPREDUCE-3060:
----------------------------------------------
+1 ! This makes a lot of optimized third party plugins possible.
> Generic shuffle service
> -----------------------
>
> Key: MAPREDUCE-3060
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3060
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Labels: shuffle
> Fix For: 0.24.0
>
>
> When I was talking to Owen about HADOOP-2600, we came across (again, talked
> about it with Chris before) the shuffle dependency issue. NodeManager
> currently has an implicit (hidden by the service plugin mechanism) dependency
> of a specific version of mapreduce shuffle. While this works in many cases,
> as long as we don't change shuffle headers and the usage of mapred security
> tokens, it's a hack to make things work none the less. It's generally agreed
> upon that nodemanager should only load generic services that are mapreduce
> framework neutral.
> In this particular case, the right solution seems to be a generic shuffle
> handler that can serve data for a particular partition securely. The
> ShuffleHandler currently only depends on mapreduce for task tokens and
> shuffle header, which is only used for writing data, i.e., the shuffle
> handler has no semantic dependency on mapreduce.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
