Hi Swetha. Finding an intuitive and easy way to inject jars/zip from multiple sources is challenging. Assuming hadoop is installed on the build box might not be feasible for all the cases. Moreover, installed hadoop version on the build box and the production version might not match. Above all, supporting multiple ways will create confustion.
Having said that, I'm not happy with the current approach either. I find the step inconvenient specially for the newbie. Let's continue this discussion for a better approach. Regards, Mohammad ________________________________ From: Shwetha GS <[email protected]> To: [email protected] Cc: "[email protected]" <[email protected]> Sent: Tuesday, September 17, 2013 10:58 PM Subject: Re: Oozie and hadoop Oh, that documentation of -hadoop option is removed now. The problem with copying hadooplibs to libext is that if the hadoop classpath has multiple directories, all jars from all directories need to be copied. Instead, if the hadoop cmd is in path, why not oozie-setup.sh automatically add all jars from 'hadoop classpath'? On Tue, Sep 17, 2013 at 7:52 PM, Alejandro Abdelnur <[email protected]>wrote: > Hi Shwetha, > > You just need to copy the JARs from your Hadoop version in libext/ and run > oozie prepare-war, you don't need to specify any version of hadoop > > http://oozie.apache.org/docs/4.0.0/DG_QuickStart.html#Server_Installation > > thx > > > On Tue, Sep 17, 2013 at 3:01 PM, Shwetha GS <[email protected]> wrote: > > > Hi, > > > > This is the way to customize oozie for specific hadoop: > > oozie-setup.sh prepare-war -hadoop HADOOP_VERSION HADOOP_PATH > > Hadoop version [0.20.1|0.20.2|0.20.104|0.20.200|0.23.x|2.x] and Hadoop > > install dir > > > > This picks specific jars according to hadoop version. This doesn't work > for > > the following: > > 1. for 0.20.2-cdh3u5 hadoop, hadoop version is 0.20.2. So, it picks just > > hadoop-core jar from hadoop path, but oozie fails with ClassNotFound > error > > for a class in guava jar. So, even guava jar should have been picked up > > 2. If I want to use hadoop 2 with MR1, none of the allowed hadoop > versions > > work for me. They either fail with jar not found or class not found > > > > Instead of picking specific jars, why not use the following logic: > > 1. if hadoop cmd is in path, add all jars in hadoop classpath cmd > > 2. else, add all jars in <HADOOP_PATH> > > > > This has advantage that it will work with any hadoop version, and oozie > > setup doesn't have to handle every possible version and its transitive > > dependencies specifically. Disadvantage is that it will make the oozie > war > > fat, but that should be ok as its built on a oozie install machine and > not > > shipped. > > > > Any thoughts? > > > > -Shwetha > > > > -- > > _____________________________________________________________ > > The information contained in this communication is intended solely for > the > > use of the individual or entity to whom it is addressed and others > > authorized to receive it. It may contain confidential or legally > privileged > > information. If you are not the intended recipient you are hereby > notified > > that any disclosure, copying, distribution or taking any action in > reliance > > on the contents of this information is strictly prohibited and may be > > unlawful. If you have received this communication in error, please notify > > us immediately by responding to this email and then delete it from your > > system. The firm is neither liable for the proper and complete > transmission > > of the information contained in this communication nor for any delay in > its > > receipt. > > > > > > -- > Alejandro > -- _____________________________________________________________ The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system. The firm is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.
