Depending on what the dependency is, you might be able to just remove it from hadoop's lib directory on your cluster.
For me, Hadoop's later versions has jackson-1.0.1 in its lib directory and that breaks usage of Avro in a M/R job among other things. However, the feature that uses this library is unimportant to me (configuration dump in JSON format) so I just removed the jar. -Scott On Apr 14, 2010, at 6:39 AM, Segel, Mike wrote: > Hi, > > Ok, here's a bit of a bizarre issue... > > How do you handle class collisions between Hadoop and your m/r job which > calls other 3rd party classes. > > An example: Hadoop has an older version of an open source jar in its /lib > directory. You're interfacing with a 3rd party OS tool that uses a later > release of the same jar. > > You can modify the classpath, and that might work. But the better way is to > create a Custom Class Loader. (Non-trivial) > > Looking at the Configuration class, it looks like there are a couple of > methods that deal with loading a class in to the configuration so that the > m/r jobs can have access to them on each node. > > Is this the correct intended use, or am I missing something? > Has anyone done something like this? > > Thx > > -Mike > > Michael Segel > Architect, R&D > NAVTEQ > 425 West Randolph Street > Chicago, IL 60606 > (T) +1 312-780-3432 > (C) +1 312-952-8175 > www.navteq.com<http://www.navteq.com/> > > > > The information contained in this communication may be CONFIDENTIAL and is > intended only for the use of the recipient(s) named above. If you are not > the intended recipient, you are hereby notified that any dissemination, > distribution, or copying of this communication, or any of its contents, is > strictly prohibited. If you have received this communication in error, > please notify the sender and delete/destroy the original message and any copy > of it from your computer or paper files.
