MRUnit driver classes should provide ability to set a configuration object to
be passed into the mapper/reducer
---------------------------------------------------------------------------------------------------------------
Key: MAPREDUCE-2002
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2002
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/mrunit
Affects Versions: 0.20.2
Reporter: David Rosenstrauch
Priority: Minor
Short description:
Enhance the org.apache.hadoop.mrunit.mapreduce.MapDriver, ReduceDriver, and
MapReduceDriver unit test driver classes to contain "setConfiguration" and
"withConfiguration" methods for passing in user-supplied
org.apache.hadoop.conf.Configuration objects, and have those configuration
objects eventually get passed on to the Context objects that are passed in to
the mapper/reducer "setup" methods. (Rather than passing in an empty
Configuration object, as is being done now.)
Long description:
The MRUnit driver classes (i.e., MapDriver, ReduceDriver, and MapReduceDriver)
ought to be enhanced to contain methods for setting a Configuration object to
be used by the mapper/reducer being tested - i.e., setConfiguration() and
withConfiguration().
The only way to effectively pass parameters into a mapper or reducer is by
setting properties on a configuration object, which the mapper/reducer can then
retrieve in their "setup" step, and use to customize its operation. As a
result, specific mappers/reducers may require the presence of specific
configuration properties/parameters in order to function correctly (or at all).
(I am currently coding such a reducer right now.)
Testing such a mapper/reducer thus requires that the unit testing framework
used provide the ability to pass in user-supplied Configuration objects to them
so that they can be tested with appropriate parameter values. However, MRUnit
currently does not provide this ability. (All mappers/reducers are always
passed an empty configuration object.) And there is not even currently any
(easy) way for the end-user to fix this problem by creating a simple sub-class
that supplies this functionality, as such subclasses would require a
substantial reimplementation/override of several MRUnit framework classes.
I believe this something that is not too difficult to fix in the MRUnit
framework code, however, and would greatly help the usability of MRUnit.
Although I don't have time to code this enhancement right now, if
needed/preferred I could squeeze out some time to code up a patch for this. If
that's needed, please let me know.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
