[
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928208#action_12928208
]
Henning Blohm commented on MAPREDUCE-1700:
------------------------------------------
The patch in MAPREDUCE-1938 does unfortunately not solve the issue when the job
implementation uses custom class loaders to load dependency classes. The
proposed patch only addresses the issue when no custom class loaders are in the
picture.
As a first step, it would really help (us anyway), if jobs would not be started
with stuff on the classpath that is not at all required for job execution per
se (e.g. jetty libs, the eclipse java compiler, jasper...).
Secondly, hadoop could actually start with only the hadoop api types on the
classpath plus a small launcher that would load hadoops implementation in an
isolated child loader, so that implementation dependencies do not leak through
to the job's implementation. I am not sure if the hadoop implementation is
ready for implementation/api separation via class loaders though.
I patched hadoop 0.20.2 to exclude all libs in lib/server from the jobs
classpath and I move all non-job related jars into that server folder in my
hadoop installation. That helped somewhat.
> User supplied dependencies may conflict with MapReduce system JARs
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1700
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: task
> Reporter: Tom White
>
> If user code has a dependency on a version of a JAR that is different to the
> one that happens to be used by Hadoop, then it may not work correctly. This
> happened with user code using a different version of Avro, as reported
> [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852081#action_12852081].
> The problem is analogous to the one that application servers have with WAR
> loading. Using a specialized classloader in the Child JVM is probably the way
> to solve this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
