Sky, Yes you are right in your summary. Thanks also for reporting back on the 0.205/1.x issue.
One ugly hack that just comes to mind, would be to manipulate the classloader at runtime. I've not tried it out to know for sure if its possible/will work, but just thought I'd note it down. However, your statement "then, there is not possible for me to use an external jar such as "commons-lang" from apache in my application. Any external jars packaged within my jar under "lib" directory are not captured." needs some refinement - You can plug and send dependent jars to a job for sure - but if that jar is already present installed on the cluster (i.e., hadoop uses it via its $HADOOP_HOME/lib/ paths), then you can't override it unless you work around. On Tue, Apr 10, 2012 at 2:16 AM, Sky USC <sky...@hotmail.com> wrote: > > > > > Thanks for the reply. I appreciate your helpfulness. I created Jars by > following instructions at "http://blog.mafr.de/2010/07/24/maven-hadoop-job/";. > So external Jars are stored in lib/ folder within a jar. > > Am I summarizing this correctly: > 1. If hadoop version = 0.20.203 or lower - then, there is not possible for me > to use an external jar such as "commons-lang" from apache in my application. > Any external jars packaged within my jar under "lib" directory are not > captured. This appears like a huge limitation to me? > 2. If hadoop version > 0.20.204 to 1.0.x - then use > "HADOOP_USER_CLASSPATH_FIRST=true" environment variable before launching > "hadoop jar" might help. I tried this for version 0.20.205 but it didnt work. > 3. If hadoop version > 2.x or formerly 0.23.x - then this can be set via API? > > Is there a working version of testable jar that has these dependencies that I > can try to figure out if its my way of packaging jar or something else?? > > Thx > >> From: ha...@cloudera.com >> Date: Mon, 9 Apr 2012 13:50:37 +0530 >> Subject: Re: How do I include the newer version of Commons-lang in my jar? >> To: common-user@hadoop.apache.org >> >> Answer is a bit messy. >> >> Perhaps you can set the environment variable "export >> HADOOP_USER_CLASSPATH_FIRST=true" before you do a "hadoop jar …" to >> launch your job. However, although this approach is present in >> 0.20.204+ (0.20.205, and 1.0.x), am not sure if it makes an impact on >> the tasks as well. I don't see it changing anything but for the driver >> CP. I've not tested it - please let us know if it works in your >> environment. >> >> In higher versions (2.x or formerly 0.23.x), this is doable from >> within your job if you set "mapreduce.job.user.classpath.first" to >> true inside your job, and ship your replacement jars along. >> >> Some versions would also let you set this via >> "JobConf/Job.setUserClassesTakesPrecedence(true/false)" API calls. >> >> On Mon, Apr 9, 2012 at 11:14 AM, Sky <sky...@hotmail.com> wrote: >> > Hi. >> > >> > I am new to Hadoop and I am working on project on AWS Elastic MapReduce. >> > >> > The problem I am facing is: >> > * org.apache.commons.lang.time.DateUtils: parseDate() works OK but >> > parseDateStrictly() fails. >> > I think parseDateStrictly might be new in lang 2.5. I thought I included >> > all >> > dependencies. However, for some reason, during runtime, my app is not >> > picking up the newer commons-lang. >> > >> > Would love some help. >> > >> > Thx >> > - sky >> > >> > >> >> >> >> -- >> Harsh J > > -- Harsh J
