hi, i need to know how to resolve conflicts with jar dependencies.
* first, my job requires Jackson JSON-processor v1.9.11. * second, the hadoop cluster has Jackson JSON-processor v1.5.2. the jars are installed in $HADOOP_HOME/lib. according to this link, http://blog.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/, there are 3 ways to include 3rd party libraries in a map/reduce (mr) job. * use the -libjars flag * include the dependent libraries in the executing jar file's /lib directory * put the jars in the $HADOOP_HOME/lib directory i can report that using -libjars and including the libraries in my jar's /lib directory "do not work" (in my case of jar conflicts). i still get a NoSuchMethodException. the only way to get my job to run is the last option, placing the newer jars in $HADOOP_HOME/lib. the last option is fine on a sandbox or development instance, but there are some political difficulties (not only technical) in modifying our production environment. my questions/concerns are: 1. how come the -libjars and /lib options do not work? how does class loading work in mr tasks? 2. is there another option available that i am not aware of to try and get dependent jars by the job to "overwrite" what's in $HADOOP_HOME/lib at runtime of the mr tasks? any help is appreciated. thank you all.
