If oozie-setup and oozie-start are done on different boxes, adding hadoop libs can be moved to oozie-start.
I agree that hadoop may not be available always, in which case it can fall back to bundled hadoop/or pick from hadoop home(if set). At the end of setup, it can print the hadoop version that it started with (which is very useful) On Wed, Sep 18, 2013 at 12:21 PM, Mohammad Islam <[email protected]> wrote: > Hi Swetha. > Finding an intuitive and easy way to inject jars/zip from multiple sources > is challenging. Assuming hadoop is installed on the build box might not be > feasible for all the cases. Moreover, installed hadoop version on the build > box and the production version might not match. > Above all, supporting multiple ways will create confustion. > > Having said that, I'm not happy with the current approach either. > > I find the step inconvenient specially for the newbie. > > Let's continue this discussion for a better approach. > > > Regards, > Mohammad > > > ________________________________ > From: Shwetha GS <[email protected]> > To: [email protected] > Cc: "[email protected]" <[email protected]> > Sent: Tuesday, September 17, 2013 10:58 PM > Subject: Re: Oozie and hadoop > > > Oh, that documentation of -hadoop option is removed now. > > The problem with copying hadooplibs to libext is that if the hadoop > classpath has multiple directories, all jars from all directories need to > be copied. Instead, if the hadoop cmd is in path, why not oozie-setup.sh > automatically add all jars from 'hadoop classpath'? > > > On Tue, Sep 17, 2013 at 7:52 PM, Alejandro Abdelnur <[email protected] > >wrote: > > > Hi Shwetha, > > > > You just need to copy the JARs from your Hadoop version in libext/ and > run > > oozie prepare-war, you don't need to specify any version of hadoop > > > > > http://oozie.apache.org/docs/4.0.0/DG_QuickStart.html#Server_Installation > > > > thx > > > > > > On Tue, Sep 17, 2013 at 3:01 PM, Shwetha GS <[email protected]> > wrote: > > > > > Hi, > > > > > > This is the way to customize oozie for specific hadoop: > > > oozie-setup.sh prepare-war -hadoop HADOOP_VERSION HADOOP_PATH > > > Hadoop version [0.20.1|0.20.2|0.20.104|0.20.200|0.23.x|2.x] and Hadoop > > > install dir > > > > > > This picks specific jars according to hadoop version. This doesn't work > > for > > > the following: > > > 1. for 0.20.2-cdh3u5 hadoop, hadoop version is 0.20.2. So, it picks > just > > > hadoop-core jar from hadoop path, but oozie fails with ClassNotFound > > error > > > for a class in guava jar. So, even guava jar should have been picked up > > > 2. If I want to use hadoop 2 with MR1, none of the allowed hadoop > > versions > > > work for me. They either fail with jar not found or class not found > > > > > > Instead of picking specific jars, why not use the following logic: > > > 1. if hadoop cmd is in path, add all jars in hadoop classpath cmd > > > 2. else, add all jars in <HADOOP_PATH> > > > > > > This has advantage that it will work with any hadoop version, and oozie > > > setup doesn't have to handle every possible version and its transitive > > > dependencies specifically. Disadvantage is that it will make the oozie > > war > > > fat, but that should be ok as its built on a oozie install machine and > > not > > > shipped. > > > > > > Any thoughts? > > > > > > -Shwetha > > > > > > -- > > > _____________________________________________________________ > > > The information contained in this communication is intended solely for > > the > > > use of the individual or entity to whom it is addressed and others > > > authorized to receive it. It may contain confidential or legally > > privileged > > > information. If you are not the intended recipient you are hereby > > notified > > > that any disclosure, copying, distribution or taking any action in > > reliance > > > on the contents of this information is strictly prohibited and may be > > > unlawful. If you have received this communication in error, please > notify > > > us immediately by responding to this email and then delete it from your > > > system. The firm is neither liable for the proper and complete > > transmission > > > of the information contained in this communication nor for any delay in > > its > > > receipt. > > > > > > > > > > > -- > > Alejandro > > > > -- > _____________________________________________________________ > The information contained in this communication is intended solely for the > use of the individual or entity to whom it is addressed and others > authorized to receive it. It may contain confidential or legally privileged > information. If you are not the intended recipient you are hereby notified > that any disclosure, copying, distribution or taking any action in reliance > on the contents of this information is strictly prohibited and may be > unlawful. If you have received this communication in error, please notify > us immediately by responding to this email and then delete it from your > system. The firm is neither liable for the proper and complete transmission > of the information contained in this communication nor for any delay in its > receipt. > -- _____________________________________________________________ The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system. The firm is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.
