[
https://issues.apache.org/jira/browse/MAPREDUCE-1574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12845180#action_12845180
]
Tom White commented on MAPREDUCE-1574:
--------------------------------------
Creating a Combiner class and _adding_ a setCombinerClass(Class<? extends
Combiner>) method, while retaining the Reducer one would not be incompatible.
However, I'm not convinced it would make things clearer, as you would still be
able to use a Reducer as a combiner - and the conditions for this would still
need explaining (only in the case that the types match, and when it makes sense
semantically for a reducer to be a combiner).
Perhaps we should be aiming for better diagnostics instead. We could improve
the error message to explain that the input and output types must match. This
could be done at the point when the framework receives a combiner output
key-value pair (or it might even be possible using an approach like
MAPREDUCE-1411).
> Combiners should implement a specialized "Combiner" interface, not the
> generic "Reducer" interface
> --------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1574
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1574
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.20.1, 0.20.2, 0.20.3
> Reporter: Danny Leshem
> Priority: Minor
>
> I just spent 30 minutes trying to figure out why my job throws
> "java.io.IOException: wrong key class" when I pass my Reducer class to
> Job.setCombinerClass. Finally, I understood that a Reducer can act as
> Combiner only if its output key/value are the same as its input key/value.
> So yes, this is documented. But you can make life easier for users by
> defining a Combiner interface (that Job.setCombinerClass will accept) to
> force this at compile time. The new interface should implement the Reducer
> interface and specialize it (is it even possible with generics?).
> Alternatively, you can call this interface "SimpleReducer".
> If the generics-trick suggested above is impossible to implement, for the
> (common?) case of having the same class acting as Combiner and Reducer you
> can do one of either:
> 1) Thin Combiner implementation that wraps a given Reducer.
> 2) Add a new method, say Job.setCombinerClassToReducer (that accepts a
> Reducer), acting similarly to the new Job.setCombinerClass - but here the
> name should alert the user she's doing something special.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
