[
https://issues.apache.org/jira/browse/MAPREDUCE-1478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom White updated MAPREDUCE-1478:
---------------------------------
Attachment: MAPREDUCE-1478.patch
move.sh
Here's an initial first cut at implementing this. The part for including the
user's classpath first is covered in MAPREDUCE-1938. This patch creates a new
source tree under src/lib for the MapReduce libraries, and the build creates a
separate library jar. It compiles, but I haven't done any testing yet. There
are a number of changes that are needed to remove core's dependency on the
libraries. Most are small, like removing dependencies on constants or
configuration names, but here are some of the other ones I made:
* Introduce InputSplitCallback so MapTask doesn't have a dependency on
FileSplit. Make mapred.FileSplit implement this interface so it can modify
JobConf before the mapper is run.
* Task depends on FileOutputCommitter. Push this code down into
FileOutputCommitter implementations of OutputCommitter#setupTask.
* MapTask, Task depend on the public WrappedMapper, WrappedReducer classes.
These need to be constructed reflectively or have private duplicates made (I
did the latter in this patch).
* org.apache.hadoop.mapreduce.util.ConfigUtil reflectively calls the new
org.apache.hadoop.mapreduce.lib.ConfigUtil so that the deprecated keys for the
libraries are added.
You need to run the move.sh script before applying the patch.
> Separate the mapred.lib and mapreduce.lib classes to a different jar and
> include the user jar ahead of the lib jar.
> -------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1478
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1478
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Reporter: Owen O'Malley
> Attachments: MAPREDUCE-1478.patch, move.sh
>
>
> Currently the user can't include updated library jars as part of their job.
> By pulling out the lib classes we can include the classes (eg.
> TextInputFormat) in the user's jar and get their version and not the system
> installed one.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
