[
https://issues.apache.org/jira/browse/MAPREDUCE-1462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12830212#action_12830212
]
Arun C Murthy commented on MAPREDUCE-1462:
------------------------------------------
{quote}
Where would these be stored? Are you proposing we add another file to each job?
Currently we have conf+splits+jar. Do we really want tasks to have to open more
files?
This proposes fundamental changes to job submission that should be addressed
elsewhere. We can achieve the goals of this issue using the existing
mechanisms, as in Tom & Aaron's patch to MAPREDUCE-1126. Changing job
submission in the way you suggest should be discussed separately, in
MAPREDUCE-1183.
{quote}
*Iff* there is consensus that this is the the model being proposed in
MAPREDUCE-1183, we could start that journey in this patch, no? Why do we need
to do the work twice i.e. first put in the conf, then move it to the
(serialized) job-description file (say, job.data) via MAPREDUCE-1183?
> Enable context-specific and stateful serializers in MapReduce
> -------------------------------------------------------------
>
> Key: MAPREDUCE-1462
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1462
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: task
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Attachments: h-1462.patch
>
>
> Although the current serializer framework is powerful, within the context of
> a job it is limited to picking a single serializer for a given class.
> Additionally, Avro generic serialization can make use of additional
> configuration/state such as the schema. (Most other serialization frameworks
> including Writable, Jute/Record IO, Thrift, Avro Specific, and Protocol
> Buffers only need the object's class name to deserialize the object.)
> With the goal of keeping the easy things easy and maintaining backwards
> compatibility, we should be able to allow applications to use context
> specific (eg. map output key) serializers in addition to the current type
> based ones that handle the majority of the cases. Furthermore, we should be
> able to support serializer specific configuration/metadata in a type safe
> manor without cluttering up the base API with a lot of new methods that will
> confuse new users.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
