[
https://issues.apache.org/jira/browse/MAPREDUCE-326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829801#action_12829801
]
Owen O'Malley commented on MAPREDUCE-326:
-----------------------------------------
I agree with Todd. I don't think taking a performance hit in Java by adding
memory copies is acceptable.
I think that keeping keys and values separate is useful for a couple of reasons:
1. keys compress better separately than intermixed with values (not that we
do that currently...)
2. combining the keys and values will effectively block using memcmp for the
sort (and it is pretty ugly to consider serializing the "key" fields
differently that the "value" fields)
3. We could support more flexible memory management if the values can be
distinguished from the keys.
4. Since Sequence Files and T Files contain both Keys and Values, they would
need to be wrapped together to present them as a single object.
I don't see what this abstraction is buying you over using ByteBuffer and a
Serializer that knows how to use it. You could define your Mapper and/or
Reducer any way you want and deserialization would be completely in the
application's hands. Very much like Google's MapReduce.
> The lowest level map-reduce APIs should be byte oriented
> --------------------------------------------------------
>
> Key: MAPREDUCE-326
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-326
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: eric baldeschwieler
>
> As discussed here:
> https://issues.apache.org/jira/browse/HADOOP-1986#action_12551237
> The templates, serializers and other complexities that allow map-reduce to
> use arbitrary types complicate the design and lead to lots of object creates
> and other overhead that a byte oriented design would not suffer. I believe
> the lowest level implementation of hadoop map-reduce should have byte string
> oriented APIs (for keys and values). This API would be more performant,
> simpler and more easily cross language.
> The existing API could be maintained as a thin layer on top of the leaner API.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
