[
https://issues.apache.org/jira/browse/HADOOP-3702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HADOOP-3702:
----------------------------------
Attachment: Hadoop-3702.patch
I am attaching a patch which is a modified version of the last one.
I have added generic arguments where applicable. But not all the instances
could be generified, since we cannot know the arguments. This is because we do
not know the map input and output types after the first map and before the last
map. At least ChainReducer and ChainMapper are now generified.
Implementing a Serializer for a specific class does not fit into the design of
Serializer. Serializer is intended to be defined once for each set of classes
(implemeting Writable, Serializable, etc.).
In the attached patch WritableJobConf extends JobConf and implements Writable.
Chain uses this class to ser/deser the configuration. And this does not
introduce an incompatibility, since Configuration/JobConf does not implement
Writable.
> add support for chaining Maps in a single Map and after a Reduce [M*/RM*]
> -------------------------------------------------------------------------
>
> Key: HADOOP-3702
> URL: https://issues.apache.org/jira/browse/HADOOP-3702
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Environment: all
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Attachments: Hadoop-3702.patch, patch3702.txt, patch3702.txt,
> patch3702.txt, patch3702.txt, patch3702.txt, patch3702.txt, patch3702.txt,
> patch3702.txt, patch3702.txt, patch3702.txt
>
>
> On the same input, we usually need to run multiple Maps one after the other
> without no Reduce. We also have to run multiple Maps after the Reduce.
> If all pre-Reduce Maps are chained together and run as a single Map a
> significant amount of Disk I/O will be avoided.
> Similarly all post-Reduce Maps can be chained together and run in the Reduce
> phase after the Reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
