[
https://issues.apache.org/jira/browse/MAPREDUCE-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-2454:
-------------------------------------
Status: Open (was: Patch Available)
Ok, I'm glad I caught this...
One issue is that you don't want to pass a shuffle down to the merger, it's the
other way around - you want to pass a merger to the shuffle.
Also, I'm not wild about making the change to MapOutput to shuffle itself, like
I said - it was initially designed as merely a 'struct'.
----
I've said this before on this jira, I'd really appreciate if you could break
this apart into smaller chunks - it makes a reviewer's job much easier... for
e.g. I missed the shuffle/merger change since it's a largish patch.
Can you pls create some natural sub-tasks:
# Move MapOutputCollector out as an interface into a new class outside of
MapTask
# Introduce a new ReduceInputMerger or some such interface which is sufficient
for your purposes
# Then mark all interfaces you need as 'LimitedPrivate'
I appreciate your patience, thanks again.
> Allow external sorter plugin for MR
> -----------------------------------
>
> Key: MAPREDUCE-2454
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2454
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Affects Versions: 2.0.2-alpha, 2.0.0-alpha, 3.0.0
> Reporter: Mariappan Asokan
> Assignee: Mariappan Asokan
> Priority: Minor
> Labels: features, performance, plugin, sort
> Attachments: HadoopSortPlugin.pdf, HadoopSortPlugin.pdf,
> KeyValueIterator.java, MapOutputSorterAbstract.java, MapOutputSorter.java,
> mapreduce-2454-modified-code.patch, mapreduce-2454-modified-test.patch,
> mapreduce-2454-new-test.patch, mapreduce-2454.patch, mapreduce-2454.patch,
> mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch,
> mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch,
> mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch,
> mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch,
> mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch,
> mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch,
> mapreduce-2454.patch, mapreduce-2454-protection-change.patch,
> mr-2454-on-mr-279-build82.patch.gz, MR-2454-trunkPatchPreview.gz,
> ReduceInputSorter.java
>
>
> Define interfaces and some abstract classes in the Hadoop framework to
> facilitate external sorter plugins both on the Map and Reduce sides.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
