[
https://issues.apache.org/jira/browse/MAPREDUCE-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13479501#comment-13479501
]
Arun C Murthy commented on MAPREDUCE-2454:
------------------------------------------
Asokan, sorry I've been busy with stuff - thanks for understanding.
I've spent sometime thinking about this - and I feel we can do something far
simpler to address Syncsort's goal of plugging in your proprietary sort while
mitigating risk to MR itself.
How about this: I feel we could accomplish both goals by something very
simple... by making MapOutputBuffer pluggable by introducing a
MapOutputCollector interface. That's about it. This way, you can supply a
custom MapOutputBuffer which plugs in your sort for your customers while we can
just keep our current implementation.
Hopefully, that makes sense. What else would you need?
I'm basically trying to vastly minimize the APIs we spread out, this way when
we want to change our sort implementation for MAPREDUCE-4039 or Sailfish etc.
we have *much* more leeway, at the same time we don't affect you at all.
Thoughts?
> Allow external sorter plugin for MR
> -----------------------------------
>
> Key: MAPREDUCE-2454
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2454
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Affects Versions: 2.0.0-alpha, 3.0.0, 2.0.2-alpha
> Reporter: Mariappan Asokan
> Assignee: Mariappan Asokan
> Priority: Minor
> Labels: features, performance, plugin, sort
> Attachments: HadoopSortPlugin.pdf, HadoopSortPlugin.pdf,
> KeyValueIterator.java, MapOutputSorterAbstract.java, MapOutputSorter.java,
> mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch,
> mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch,
> mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch,
> mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch,
> mr-2454-on-mr-279-build82.patch.gz, MR-2454-trunkPatchPreview.gz,
> ReduceInputSorter.java
>
>
> Define interfaces and some abstract classes in the Hadoop framework to
> facilitate external sorter plugins both on the Map and Reduce sides.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
