I agree that it writing a custom classloader is non-trivial (and I don't know if anyone has done this in the MapReduce context), so this would best be handled by the framework. I've opened https://issues.apache.org/jira/browse/MAPREDUCE-1700 for this.
Tom On Wed, Apr 14, 2010 at 6:39 AM, Segel, Mike <[email protected]> wrote: > Hi, > > Ok, here's a bit of a bizarre issue... > > How do you handle class collisions between Hadoop and your m/r job which > calls other 3rd party classes. > > An example: Hadoop has an older version of an open source jar in its /lib > directory. You're interfacing with a 3rd party OS tool that uses a later > release of the same jar. > > You can modify the classpath, and that might work. But the better way is to > create a Custom Class Loader. (Non-trivial) > > Looking at the Configuration class, it looks like there are a couple of > methods that deal with loading a class in to the configuration so that the > m/r jobs can have access to them on each node. > > Is this the correct intended use, or am I missing something? > Has anyone done something like this? > > Thx > > -Mike > > Michael Segel > Architect, R&D > NAVTEQ > 425 West Randolph Street > Chicago, IL 60606 > (T) +1 312-780-3432 > (C) +1 312-952-8175 > www.navteq.com<http://www.navteq.com/> > > > > The information contained in this communication may be CONFIDENTIAL and is > intended only for the use of the recipient(s) named above. If you are not > the intended recipient, you are hereby notified that any dissemination, > distribution, or copying of this communication, or any of its contents, is > strictly prohibited. If you have received this communication in error, > please notify the sender and delete/destroy the original message and any copy > of it from your computer or paper files. >
