[
https://issues.apache.org/jira/browse/MAPREDUCE-3060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated MAPREDUCE-3060:
-------------------------------
Target Version/s: (was: 0.24.0)
> Generic shuffle service
> -----------------------
>
> Key: MAPREDUCE-3060
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3060
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Luke Lu
> Labels: shuffle
>
> When I was talking to Owen about MAPREDUCE-2600, we came across (again,
> talked about it with Chris before) the shuffle dependency issue. NodeManager
> currently has an implicit (hidden by the service plugin mechanism) dependency
> of a specific version of mapreduce shuffle. While this works in many cases,
> as long as we don't change shuffle headers and the usage of mapred security
> tokens, it's a hack to make things work none the less. It's generally agreed
> upon that nodemanager should only load generic services that are mapreduce
> framework neutral.
> In this particular case, the right solution seems to be a generic shuffle
> handler that can serve data for a particular partition securely. The
> ShuffleHandler currently only depends on mapreduce for task tokens and
> shuffle header, which is only used for writing data, i.e., the shuffle
> handler has no semantic dependency on mapreduce.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
