I agree with the proposal. As Mohammad mentioned as well, users are finding the current approach counter-intuitive. Recently Bowen faced some trouble injecting the right Hcatalog Jar, and if already in class path, it would alleviate the need to know list of such add-on JARs for a quick newbie deployment.
On 9/19/13 4:19 AM, "Alejandro Abdelnur" <[email protected]> wrote: >The current model, all JARs in the WAR follows the J2EE deployment model. >An app (the WAR in this case) is self-contained. My J2EE background can be >blamed from this choice. > >While I like this model, in practice it seems people stumble up this quite >often. > >Maybe we should switch to a model closer to what Bigtop does, no >bundling/injecting Hadoop JARs, instead adding them to the classpath of >Tomcat, and the same for all the Oozie JARs. The WAR will have not classes >(you want all JARs in the same classloader). > >The only challenge here is that the Hadoop JARs available in the classpath >of the Hadoop client have tons of stuff that will break Ooize server >(servlet, jsp, jasper, jetty, .... JARs). We should first get a Hadoop >client with the 'client' JARs, then we could do that. > >Said this, this would work from version of Hadoop that do this onwards. > >Thx > > >On Wed, Sep 18, 2013 at 11:20 AM, Shwetha GS <[email protected]> >wrote: > >> If oozie-setup and oozie-start are done on different boxes, adding >>hadoop >> libs can be moved to oozie-start. >> >> I agree that hadoop may not be available always, in which case it can >>fall >> back to bundled hadoop/or pick from hadoop home(if set). At the end of >> setup, it can print the hadoop version that it started with (which is >>very >> useful) >> >> >> On Wed, Sep 18, 2013 at 12:21 PM, Mohammad Islam <[email protected]> >> wrote: >> >> > Hi Swetha. >> > Finding an intuitive and easy way to inject jars/zip from multiple >> sources >> > is challenging. Assuming hadoop is installed on the build box might >>not >> be >> > feasible for all the cases. Moreover, installed hadoop version on the >> build >> > box and the production version might not match. >> > Above all, supporting multiple ways will create confustion. >> > >> > Having said that, I'm not happy with the current approach either. >> > >> > I find the step inconvenient specially for the newbie. >> > >> > Let's continue this discussion for a better approach. >> > >> > >> > Regards, >> > Mohammad >> > >> > >> > ________________________________ >> > From: Shwetha GS <[email protected]> >> > To: [email protected] >> > Cc: "[email protected]" >><[email protected] >> > >> > Sent: Tuesday, September 17, 2013 10:58 PM >> > Subject: Re: Oozie and hadoop >> > >> > >> > Oh, that documentation of -hadoop option is removed now. >> > >> > The problem with copying hadooplibs to libext is that if the hadoop >> > classpath has multiple directories, all jars from all directories >>need to >> > be copied. Instead, if the hadoop cmd is in path, why not >>oozie-setup.sh >> > automatically add all jars from 'hadoop classpath'? >> > >> > >> > On Tue, Sep 17, 2013 at 7:52 PM, Alejandro Abdelnur <[email protected] >> > >wrote: >> > >> > > Hi Shwetha, >> > > >> > > You just need to copy the JARs from your Hadoop version in libext/ >>and >> > run >> > > oozie prepare-war, you don't need to specify any version of hadoop >> > > >> > > >> > >> >>http://oozie.apache.org/docs/4.0.0/DG_QuickStart.html#Server_Installation >> > > >> > > thx >> > > >> > > >> > > On Tue, Sep 17, 2013 at 3:01 PM, Shwetha GS <[email protected]> >> > wrote: >> > > >> > > > Hi, >> > > > >> > > > This is the way to customize oozie for specific hadoop: >> > > > oozie-setup.sh prepare-war -hadoop HADOOP_VERSION HADOOP_PATH >> > > > Hadoop version [0.20.1|0.20.2|0.20.104|0.20.200|0.23.x|2.x] and >> Hadoop >> > > > install dir >> > > > >> > > > This picks specific jars according to hadoop version. This doesn't >> work >> > > for >> > > > the following: >> > > > 1. for 0.20.2-cdh3u5 hadoop, hadoop version is 0.20.2. So, it >>picks >> > just >> > > > hadoop-core jar from hadoop path, but oozie fails with >>ClassNotFound >> > > error >> > > > for a class in guava jar. So, even guava jar should have been >>picked >> up >> > > > 2. If I want to use hadoop 2 with MR1, none of the allowed hadoop >> > > versions >> > > > work for me. They either fail with jar not found or class not >>found >> > > > >> > > > Instead of picking specific jars, why not use the following logic: >> > > > 1. if hadoop cmd is in path, add all jars in hadoop classpath cmd >> > > > 2. else, add all jars in <HADOOP_PATH> >> > > > >> > > > This has advantage that it will work with any hadoop version, and >> oozie >> > > > setup doesn't have to handle every possible version and its >> transitive >> > > > dependencies specifically. Disadvantage is that it will make the >> oozie >> > > war >> > > > fat, but that should be ok as its built on a oozie install machine >> and >> > > not >> > > > shipped. >> > > > >> > > > Any thoughts? >> > > > >> > > > -Shwetha >> > > > >> > > > -- >> > > > _____________________________________________________________ >> > > > The information contained in this communication is intended solely >> for >> > > the >> > > > use of the individual or entity to whom it is addressed and others >> > > > authorized to receive it. It may contain confidential or legally >> > > privileged >> > > > information. If you are not the intended recipient you are hereby >> > > notified >> > > > that any disclosure, copying, distribution or taking any action in >> > > reliance >> > > > on the contents of this information is strictly prohibited and >>may be >> > > > unlawful. If you have received this communication in error, please >> > notify >> > > > us immediately by responding to this email and then delete it from >> your >> > > > system. The firm is neither liable for the proper and complete >> > > transmission >> > > > of the information contained in this communication nor for any >>delay >> in >> > > its >> > > > receipt. >> > > > >> > > >> > > >> > > >> > > -- >> > > Alejandro >> > > >> > >> > -- >> > _____________________________________________________________ >> > The information contained in this communication is intended solely for >> the >> > use of the individual or entity to whom it is addressed and others >> > authorized to receive it. It may contain confidential or legally >> privileged >> > information. If you are not the intended recipient you are hereby >> notified >> > that any disclosure, copying, distribution or taking any action in >> reliance >> > on the contents of this information is strictly prohibited and may be >> > unlawful. If you have received this communication in error, please >>notify >> > us immediately by responding to this email and then delete it from >>your >> > system. The firm is neither liable for the proper and complete >> transmission >> > of the information contained in this communication nor for any delay >>in >> its >> > receipt. >> > >> >> -- >> _____________________________________________________________ >> The information contained in this communication is intended solely for >>the >> use of the individual or entity to whom it is addressed and others >> authorized to receive it. It may contain confidential or legally >>privileged >> information. If you are not the intended recipient you are hereby >>notified >> that any disclosure, copying, distribution or taking any action in >>reliance >> on the contents of this information is strictly prohibited and may be >> unlawful. If you have received this communication in error, please >>notify >> us immediately by responding to this email and then delete it from your >> system. The firm is neither liable for the proper and complete >>transmission >> of the information contained in this communication nor for any delay in >>its >> receipt. >> > > > >-- >Alejandro
