Iván de Prado created MAPREDUCE-4876:
----------------------------------------
Summary: Adopt a Tuple MapReduce API instead of classic MapReduce
one
Key: MAPREDUCE-4876
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4876
Project: Hadoop Map/Reduce
Issue Type: Wish
Reporter: Iván de Prado
Priority: Minor
After using MapReduce for many years, we have notice that it lacks some
important features: compound records, easy intra-reduce sorting and join
capabilities. We have elaborated slightly modified MapReduce foundation to
overcome these problems: Tuple MapReduce. You can see a full paper published at
the ICDM 2012 that describes it at http://pangool.net/TupleMapReduce.pdf
The good news are:
1) That it is not needed any architectural change on Hadoop to embrace Tuple
MapReduce
2) Indeed, we have proven that it is possible to implement it on top of Hadoop.
See the Pangool Open Source project ( http://pangool.net/ ).
3) It performs very efficiently ( http://pangool.net/benchmark.html )
4) It is compatible with all Hadoop stack: Writables, Serializers,
Input/OutputFormats, etc.
We believe Hadoop community could benefits from it in different ways:
1) By getting ideas for future API redesign
2) By adopting Pangool inside Hadoop. Of course, we would be helping and
contributing with anything needed, including by doing any adaptation changes
needed (too few, because as I told, everything is compatible with existing
MapReduce)
Obviously, we prefer the second. But at least, we believe some good ideas can
be obtained by looking at Tuple MapReduce and Pangool.
There are also other improvements in Pangool that would improve Hadoop API:
1) Configuration by instance: passing parameters by constructor. For example,
Pangool Input/OutputFormats can be configured by providing values to the
constructor
2) Stateful serialization. What is requested in
https://issues.apache.org/jira/browse/MAPREDUCE-1462 is supported by Pangool
3) First-class multipleinput/multipleoutput
Well, we are open to the discussion and to contribute.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
