[
https://issues.apache.org/jira/browse/HADOOP-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12584986#action_12584986
]
Runping Qi commented on HADOOP-3149:
------------------------------------
Alejandro,
Having a private mos in mapper/reducer class, in additional the standard
collector passed in through map/reduce call is really urgly and is redudant,
and is not compatible with the common patterns of applications using
map/reduce framework.
All the output dispatching logic should be handled by the class of collecor
argument.
That is exactly why MultipleOutputFormat classes were introduced. You may think
it is not obvious at first. But like many of the other aspects of
map/reduce framework, you will get used to it, and once you get used to it,
you'll find it very natural.
I found it is most common to that all the output key values are of the same
types.
In case you need to write keys/values of different types, you may need to
encapsulate them in a container class and use
the container class as the key (value) class for the file. The
MultipleOutputFormat allows you to write key/value of different types to
different files.
> supporting multiple outputs for M/R jobs
> ----------------------------------------
>
> Key: HADOOP-3149
> URL: https://issues.apache.org/jira/browse/HADOOP-3149
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Environment: all
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 0.17.0
>
> Attachments: patch3149.txt, patch3149.txt, patch3149.txt,
> patch3149.txt
>
>
> The outputcollector supports writing data to a single output, the 'part'
> files in the output path.
> We found quite common that our M/R jobs have to write data to different
> output. For example when classifying data as NEW, UPDATE, DELETE, NO-CHANGE
> to later do different processing on it.
> Handling the initialization of additional outputs from within the M/R code
> complicates the code and is counter intuitive with the notion of job
> configuration.
> It would be desirable to:
> # Configure the additional outputs in the jobconf, potentially specifying
> different outputformats, key and value classes for each one.
> # Write to the additional outputs in a similar way as data is written to the
> outputcollector.
> # Support the speculative execution semantics for the output files, only
> visible in the final output for promoted tasks.
> To support multiple outputs the following classes would be added to
> mapred/lib:
> * {{MOJobConf}} : extends {{JobConf}} adding methods to define named outputs
> (name, outputformat, key class, value class)
> * {{MOOutputCollector}} : extends {{OutputCollector}} adding a
> {{collect(String outputName, WritableComparable key, Writable value)}} method.
> * {{MOMapper}} and {{MOReducer}}: implement {{Mapper}} and {{Reducer}} adding
> a new {{configure}}, {{map}} and {{reduce}} signature that take the
> corresponding {{MO}} classes and performs the proper initialization.
> The data flow behavior would be: key/values written to the default (unnamed)
> output (using the original OutputCollector {{collect}} signature) take part
> of the shuffle/sort/reduce processing phases. key/values written to a named
> output from within a map don't.
> The named output files would be named using the task type and task ID to
> avoid collision among tasks (i.e. 'new-m-00002' and 'new-r-00001').
> Together with the setInputPathFilter feature introduced by HADOOP-2055 it
> would become very easy to chain jobs working on particular named outputs
> within a single directory.
> We are using heavily this pattern and it greatly simplified our M/R code as
> well as chaining different M/R.
> We wanted to contribute this back to Hadoop as we think is a generic feature
> many could benefit from.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
