[
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857076#action_12857076
]
Arun C Murthy commented on MAPREDUCE-1700:
------------------------------------------
bq. If the framework could support installing jars into an 'application'
classloader space and then jobs reference that space, task latency could be
reduced significantly as each job submission would not need to also submit all
its dependency jars.
Scott, this is precisely what the DistributedCache was designed for. Please
load your jars to HDFS, add your jars to the DistributedCache and then they are
'localized' once per-tasktracker and all jobs can use the same:
http://hadoop.apache.org/common/docs/r0.20.0/api/org/apache/hadoop/filecache/DistributedCache.html
> User supplied dependencies may conflict with MapReduce system JARs
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1700
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: task
> Reporter: Tom White
>
> If user code has a dependency on a version of a JAR that is different to the
> one that happens to be used by Hadoop, then it may not work correctly. This
> happened with user code using a different version of Avro, as reported
> [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852081#action_12852081].
> The problem is analogous to the one that application servers have with WAR
> loading. Using a specialized classloader in the Child JVM is probably the way
> to solve this.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
