The current model, all JARs in the WAR follows the J2EE deployment model. An app (the WAR in this case) is self-contained. My J2EE background can be blamed from this choice.
While I like this model, in practice it seems people stumble up this quite often. Maybe we should switch to a model closer to what Bigtop does, no bundling/injecting Hadoop JARs, instead adding them to the classpath of Tomcat, and the same for all the Oozie JARs. The WAR will have not classes (you want all JARs in the same classloader). The only challenge here is that the Hadoop JARs available in the classpath of the Hadoop client have tons of stuff that will break Ooize server (servlet, jsp, jasper, jetty, .... JARs). We should first get a Hadoop client with the 'client' JARs, then we could do that. Said this, this would work from version of Hadoop that do this onwards. Thx On Wed, Sep 18, 2013 at 11:20 AM, Shwetha GS <[email protected]> wrote: > If oozie-setup and oozie-start are done on different boxes, adding hadoop > libs can be moved to oozie-start. > > I agree that hadoop may not be available always, in which case it can fall > back to bundled hadoop/or pick from hadoop home(if set). At the end of > setup, it can print the hadoop version that it started with (which is very > useful) > > > On Wed, Sep 18, 2013 at 12:21 PM, Mohammad Islam <[email protected]> > wrote: > > > Hi Swetha. > > Finding an intuitive and easy way to inject jars/zip from multiple > sources > > is challenging. Assuming hadoop is installed on the build box might not > be > > feasible for all the cases. Moreover, installed hadoop version on the > build > > box and the production version might not match. > > Above all, supporting multiple ways will create confustion. > > > > Having said that, I'm not happy with the current approach either. > > > > I find the step inconvenient specially for the newbie. > > > > Let's continue this discussion for a better approach. > > > > > > Regards, > > Mohammad > > > > > > ________________________________ > > From: Shwetha GS <[email protected]> > > To: [email protected] > > Cc: "[email protected]" <[email protected] > > > > Sent: Tuesday, September 17, 2013 10:58 PM > > Subject: Re: Oozie and hadoop > > > > > > Oh, that documentation of -hadoop option is removed now. > > > > The problem with copying hadooplibs to libext is that if the hadoop > > classpath has multiple directories, all jars from all directories need to > > be copied. Instead, if the hadoop cmd is in path, why not oozie-setup.sh > > automatically add all jars from 'hadoop classpath'? > > > > > > On Tue, Sep 17, 2013 at 7:52 PM, Alejandro Abdelnur <[email protected] > > >wrote: > > > > > Hi Shwetha, > > > > > > You just need to copy the JARs from your Hadoop version in libext/ and > > run > > > oozie prepare-war, you don't need to specify any version of hadoop > > > > > > > > > http://oozie.apache.org/docs/4.0.0/DG_QuickStart.html#Server_Installation > > > > > > thx > > > > > > > > > On Tue, Sep 17, 2013 at 3:01 PM, Shwetha GS <[email protected]> > > wrote: > > > > > > > Hi, > > > > > > > > This is the way to customize oozie for specific hadoop: > > > > oozie-setup.sh prepare-war -hadoop HADOOP_VERSION HADOOP_PATH > > > > Hadoop version [0.20.1|0.20.2|0.20.104|0.20.200|0.23.x|2.x] and > Hadoop > > > > install dir > > > > > > > > This picks specific jars according to hadoop version. This doesn't > work > > > for > > > > the following: > > > > 1. for 0.20.2-cdh3u5 hadoop, hadoop version is 0.20.2. So, it picks > > just > > > > hadoop-core jar from hadoop path, but oozie fails with ClassNotFound > > > error > > > > for a class in guava jar. So, even guava jar should have been picked > up > > > > 2. If I want to use hadoop 2 with MR1, none of the allowed hadoop > > > versions > > > > work for me. They either fail with jar not found or class not found > > > > > > > > Instead of picking specific jars, why not use the following logic: > > > > 1. if hadoop cmd is in path, add all jars in hadoop classpath cmd > > > > 2. else, add all jars in <HADOOP_PATH> > > > > > > > > This has advantage that it will work with any hadoop version, and > oozie > > > > setup doesn't have to handle every possible version and its > transitive > > > > dependencies specifically. Disadvantage is that it will make the > oozie > > > war > > > > fat, but that should be ok as its built on a oozie install machine > and > > > not > > > > shipped. > > > > > > > > Any thoughts? > > > > > > > > -Shwetha > > > > > > > > -- > > > > _____________________________________________________________ > > > > The information contained in this communication is intended solely > for > > > the > > > > use of the individual or entity to whom it is addressed and others > > > > authorized to receive it. It may contain confidential or legally > > > privileged > > > > information. If you are not the intended recipient you are hereby > > > notified > > > > that any disclosure, copying, distribution or taking any action in > > > reliance > > > > on the contents of this information is strictly prohibited and may be > > > > unlawful. If you have received this communication in error, please > > notify > > > > us immediately by responding to this email and then delete it from > your > > > > system. The firm is neither liable for the proper and complete > > > transmission > > > > of the information contained in this communication nor for any delay > in > > > its > > > > receipt. > > > > > > > > > > > > > > > > -- > > > Alejandro > > > > > > > -- > > _____________________________________________________________ > > The information contained in this communication is intended solely for > the > > use of the individual or entity to whom it is addressed and others > > authorized to receive it. It may contain confidential or legally > privileged > > information. If you are not the intended recipient you are hereby > notified > > that any disclosure, copying, distribution or taking any action in > reliance > > on the contents of this information is strictly prohibited and may be > > unlawful. If you have received this communication in error, please notify > > us immediately by responding to this email and then delete it from your > > system. The firm is neither liable for the proper and complete > transmission > > of the information contained in this communication nor for any delay in > its > > receipt. > > > > -- > _____________________________________________________________ > The information contained in this communication is intended solely for the > use of the individual or entity to whom it is addressed and others > authorized to receive it. It may contain confidential or legally privileged > information. If you are not the intended recipient you are hereby notified > that any disclosure, copying, distribution or taking any action in reliance > on the contents of this information is strictly prohibited and may be > unlawful. If you have received this communication in error, please notify > us immediately by responding to this email and then delete it from your > system. The firm is neither liable for the proper and complete transmission > of the information contained in this communication nor for any delay in its > receipt. > -- Alejandro
