Oh, we can use trunk as Oozie 4.0. In that case I would leave the default as true (to use the launcher jar) in trunk and create a JIRA to change it to false for 5.0. Easier for me if we just use trunk :)
thanks - Robert On Thu, Jun 6, 2013 at 10:54 AM, Rohini Palaniswamy <rohini.adi...@gmail.com > wrote: > Robert, > Do you want to spend effort to rebase and put this patch in branch-4 > also? I was wondering since we were thinking of releasing trunk with SLA > work along with HCat as 4.0 instead of releasing the current branch-4 to > consolidate the database changes. If you would like to release branch-4, > then we need to do the OozieDBCLI table changes for SLA work as 5.0 and > change trunk to 5.0 then. > > > Regards, > Rohini > > > On Thu, Jun 6, 2013 at 10:04 AM, Robert Kanter <rkan...@cloudera.com> > wrote: > > > It took forever, but I finally got the refactoring done; I just uploaded > a > > patch at OOZIE-1315 <https://issues.apache.org/jira/browse/OOZIE-1315>. > > Please take a look. > > > > The patch has the default set to true (use the launcher jar). When I > > commit it I can have the version in branch-4 use true and the version in > > trunk use false. > > > > thanks > > - Robert > > > > > > On Mon, May 6, 2013 at 3:06 PM, Rohini Palaniswamy > > <rohini.adi...@gmail.com>wrote: > > > > > Tucu, > > > That sounds good. Can we have the changes go in trunk first and once > > we > > > validate that everything is fine, can we then port all the patches to > > > branch-4.0? > > > > > > Thanks, > > > Rohini > > > > > > > > > On Mon, May 6, 2013 at 2:52 PM, Alejandro Abdelnur <t...@cloudera.com > > > >wrote: > > > > > > > Rohini, > > > > > > > > I understand you don't want to destabilize Oozie 4. Still, as we have > > no > > > > done a 4 release yet, I think it is a good time to introduce this > > > feature. > > > > > > > > How about the following (similar to what we did for command > > > interruptions)? > > > > > > > > We do as Robert proposed in his last email, but keeping the default > > value > > > > on OFF for Oozie 4; and in Oozie 5 we invert the default value to ON. > > > > > > > > This would mean you don't have to change anything in your deployment > > > > protocol for Oozie 4. > > > > > > > > Thx > > > > > > > > > > > > > > > > On Mon, May 6, 2013 at 2:30 PM, Rohini Palaniswamy < > > > > rohini.adi...@gmail.com> wrote: > > > > > > > >> Robert, > > > >> > A new major release, like Oozie 4, would be a good time to > introduce > > > >> this change as the default; > > > >> > > > >> We already have Oozie 4 deployed in some clusters and it is > stable > > > >> now. We are waiting for users to validate Oozie-HCat integration > > before > > > we > > > >> deploy them in production clusters and call out a release in Apache. > > So > > > it > > > >> would be good to introduce this in the next release which will also > > be a > > > >> big release anyway. > > > >> > > > >> Regards, > > > >> Rohini > > > >> > > > >> > > > >> On Mon, May 6, 2013 at 2:09 PM, Robert Kanter <rkan...@cloudera.com > > > >wrote: > > > >> > > > >>> Sorry, I realize that I wasn't clear on that point. The Main > classes > > > >>> would > > > >>> go in their respective sharelib (e.g. PigMain goes in > share/lib/pig/) > > > but > > > >>> the other few classes that all actions use in the launcher jar > (e.g. > > > >>> LauncherMapper) would go in the oozie sharelib (share/lib/oozie), > > which > > > >>> IIRC is already included when you use sharelibs (for example, when > > you > > > >>> use > > > >>> the sharelib with the Pig action, it includes both the > share/lib/pig > > > and > > > >>> share/lib/oozie). > > > >>> > > > >>> thanks > > > >>> - Robert > > > >>> > > > >>> > > > >>> On Mon, May 6, 2013 at 1:52 PM, Virag Kothari <vi...@yahoo-inc.com > > > > > >>> wrote: > > > >>> > > > >>> > Hi Robert, > > > >>> > > > > >>> > I didn't understand completely. Are you suggesting to have all > > > action > > > >>> > specific main classes in /share/lib/oozie instead of their > > respective > > > >>> > action sharelibs? > > > >>> > > > > >>> > Thanks, > > > >>> > Virag > > > >>> > > > > >>> > From: Robert Kanter <rkan...@cloudera.com> > > > >>> > Date: Monday, May 6, 2013 12:47 PM > > > >>> > To: "dev@oozie.apache.org" <dev@oozie.apache.org>, Virag > Kothari < > > > >>> > vi...@yahoo-inc.com>, Alejandro Abdelnur <t...@cloudera.com> > > > >>> > Subject: Re: OOZIE-1311 > > > >>> > > > > >>> > Virag, > > > >>> > I talked more with Alejandro about the sharelib and the launcher > > jar. > > > >>> > > > > >>> > It would be good if we could eventually get rid of the launcher > jar > > > >>> > completely because: > > > >>> > - less files would be copied to HDFS so it should be faster > > > >>> > - it would fix an issue where some linux distress cleanup the > temp > > > dir > > > >>> and > > > >>> > delete the launcher jar > > > >>> > > > > >>> > The contents of the launcher jar could be moved to the Oozie > > > sharelib > > > >>> > (/share/lib/oozie) which IIRC currently only has a json jar file. > > > >>> > > > > >>> > As for backwards compatibility, we could do something like what > > you > > > >>> > suggested for the other sharelibs where the webapp module > includes > > > >>> them. > > > >>> > Then we'd add some property to oozie-site that would switch > > between > > > >>> the > > > >>> > old behavior with the launcher jar (sharelib is not required) and > > the > > > >>> new > > > >>> > behavior without the launcher jar (sharelib is required). A new > > > major > > > >>> > release, like Oozie 4, would be a good time to introduce this > > change > > > >>> as the > > > >>> > default; users can use the property to go back to the old > behavior. > > > >>> Also, > > > >>> > if set to use the new behavior, we should make > > > >>> oozie.use.system.libpath in > > > >>> > the job.properties default to true because the sharelib would be > > > >>> required, > > > >>> > and if set to the old behavior, it would default to false. > > > >>> > > > > >>> > Thoughts? > > > >>> > > > > >>> > thanks > > > >>> > - Robert > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > On Wed, Apr 24, 2013 at 12:24 PM, Virag Kothari < > > vi...@yahoo-inc.com > > > >>> >wrote: > > > >>> > > > > >>> >> Thanks Robert. Created OOZIE-1341 > > > >>> >> > > > >>> >> On 4/24/13 11:36 AM, "Robert Kanter" <rkan...@cloudera.com> > > wrote: > > > >>> >> > > > >>> >> >I think I follow what you're saying; assuming everything works > > > >>> correctly, > > > >>> >> >I > > > >>> >> >think that sounds good. Can you make a JIRA with that and put > > up a > > > >>> >> patch; > > > >>> >> >that might be easier to see how this would work. > > > >>> >> > > > > >>> >> >thanks > > > >>> >> >- Robert > > > >>> >> > > > > >>> >> > > > > >>> >> >On Wed, Apr 24, 2013 at 11:29 AM, Virag Kothari < > > > vi...@yahoo-inc.com > > > >>> > > > > >>> >> >wrote: > > > >>> >> > > > > >>> >> >> We can keep the Main classes in the share lib, but have the > > > action > > > >>> >> >> executors package the main classes for shipping it to > > > >>> >> >> Launcher (basically again having the method > > getLauncherClasses() > > > in > > > >>> >> >>action > > > >>> >> >> executors). Also the oozie-sharelib.jar containing main > classes > > > >>> >> >> should be part of web-app. > > > >>> >> >> This will improve the building/testing env as classes are in > > > >>> isolated > > > >>> >> >> sharelibs but will make the share lib optional as main > classes > > > are > > > >>> part > > > >>> >> >>of > > > >>> >> >> oozie-server. > > > >>> >> >> Can we do this change to trunk as this is blocking our QE's > to > > > pick > > > >>> >> >>builds > > > >>> >> >> due to share lib incompatibility? > > > >>> >> >> > > > >>> >> >> We can still do what oozie-1318 is trying to achieve. For, > e.g > > > with > > > >>> >> Hcat > > > >>> >> >> integration, > > > >>> >> >> there can be multiple overridable implementation of > URIHandlers > > > >>> that > > > >>> >> can > > > >>> >> >> be shipped to launcher. > > > >>> >> >> We can do the same for other main classes if required. > > > >>> >> >> > > > >>> >> >> > > > >>> >> >> Thanks, > > > >>> >> >> Virag > > > >>> >> >> > > > >>> >> >> > > > >>> >> >> On 4/24/13 10:18 AM, "Robert Kanter" <rkan...@cloudera.com> > > > wrote: > > > >>> >> >> > > > >>> >> >> >That's true, its primarily a building/testing improvement. > > And > > > >>> from a > > > >>> >> >> >design perspective, it could be considered "cleaner" to not > > have > > > >>> the > > > >>> >> >> >launcher jar. > > > >>> >> >> > > > > >>> >> >> >Your idea sounds like a good compromise between backwards > > > >>> >> compatibility > > > >>> >> >> >and > > > >>> >> >> >improving the building/testing env. How would we do that? > > > >>> >> >> > > > > >>> >> >> >As a side note, there's also a new feature with the Main > > classes > > > >>> being > > > >>> >> >>in > > > >>> >> >> >the sharelib; users can now override the Main class that > Oozie > > > >>> will > > > >>> >> use > > > >>> >> >> >for > > > >>> >> >> >the action because its no longer in the launcher jar. I > think > > > >>> this > > > >>> >> >>could > > > >>> >> >> >still be a useful feature, but I'm guessing that if we do > the > > > >>> >> >>compromise, > > > >>> >> >> >this would go away? In any case, OOZIE-1318 is to make the > > Main > > > >>> >> >> >overridable with a config, which is probably cleaner and > more > > > >>> flexible > > > >>> >> >> >than > > > >>> >> >> >replacing a jar file. > > > >>> >> >> > > > > >>> >> >> >thanks > > > >>> >> >> >- Robert > > > >>> >> >> > > > > >>> >> >> > > > > >>> >> >> >On Wed, Apr 24, 2013 at 9:54 AM, Virag Kothari < > > > >>> vi...@yahoo-inc.com> > > > >>> >> >> >wrote: > > > >>> >> >> > > > > >>> >> >> >> That is a good suggestion for preserving compatibility, > > > Robert. > > > >>> >> >> >> > > > >>> >> >> >> Another question is what is the main purpose of > OOZIE-1311? > > > From > > > >>> >> JIRA > > > >>> >> >> >> description, the primary > > > >>> >> >> >> advantage of moving Main classes from core to share lib is > > it > > > >>> can > > > >>> >> >>help > > > >>> >> >> >>in > > > >>> >> >> >> preventing dependencies conflicts > > > >>> >> >> >> (different versions of antlr's for pig, hive). However, > that > > > >>> problem > > > >>> >> >> >>only > > > >>> >> >> >> exists while building/testing Oozie. > > > >>> >> >> >> If that is the case, main classes can be moved to share > lib > > > but > > > >>> they > > > >>> >> >>can > > > >>> >> >> >> be bundled in oozie-server itself. > > > >>> >> >> >> In that case, the deployment doesn't change and share lib > > can > > > >>> still > > > >>> >> >>be > > > >>> >> >> >> optional > > > >>> >> >> >> as oozie server would ship those classes (same as what was > > > >>> happening > > > >>> >> >> >> before). And, > > > >>> >> >> >> advantage of 1311 for building/testing will be retained > > > >>> >> >> >> > > > >>> >> >> >> The main concern is having share lib as a required > > > installation > > > >>> will > > > >>> >> >> >> complicate deployment. > > > >>> >> >> >> Doing hot upgrade for share lib is non-trivial and Oozie > > > doesn't > > > >>> >> >>provide > > > >>> >> >> >> inbuilt support for that. > > > >>> >> >> >> Even though end-users wouldn't notice the change, it will > > be a > > > >>> big > > > >>> >> >> >>change > > > >>> >> >> >> for system admins installing oozie > > > >>> >> >> >> So even though share lib is nice feature to have, keeping > it > > > >>> >> optional > > > >>> >> >> >> seems better to me as there are other ways > > > >>> >> >> >> Of including jars and which were introduced much before > > share > > > >>> lib. > > > >>> >> >> >> > > > >>> >> >> >> Thanks, > > > >>> >> >> >> Virag > > > >>> >> >> >> > > > >>> >> >> >> > > > >>> >> >> >> > > > >>> >> >> >> > > > >>> >> >> >> On 4/24/13 9:09 AM, "Robert Kanter" <rkan...@cloudera.com > > > > > >>> wrote: > > > >>> >> >> >> > > > >>> >> >> >> >Alejandro had another suggestion to prevent breaking > stuff > > > for > > > >>> >> >>existing > > > >>> >> >> >> >apps where the sharelib isn't used. Assuming the > sharelib > > is > > > >>> >> >> >>installed, > > > >>> >> >> >> >if > > > >>> >> >> >> >the user doesn't set oozie.use.system.libpath to true, > then > > > >>> Oozie > > > >>> >> >>would > > > >>> >> >> >> >only include the oozie-*.jar from the sharelib (the one > > with > > > >>> the > > > >>> >> >>Main) > > > >>> >> >> >>in > > > >>> >> >> >> >it; if its set to true, then Oozie would include all of > the > > > >>> jars in > > > >>> >> >>the > > > >>> >> >> >> >sharelib. This would allow users to still use their own > > lib > > > >>> dir > > > >>> >> >>while > > > >>> >> >> >> >still getting the Main into the classpath. > > > >>> >> >> >> > > > > >>> >> >> >> >- Robert > > > >>> >> >> >> > > > > >>> >> >> >> > > > > >>> >> >> >> > > > > >>> >> >> >> >On Tue, Apr 23, 2013 at 10:46 PM, Alejandro Abdelnur > > > >>> >> >> >> ><tuc...@gmail.com>wrote: > > > >>> >> >> >> > > > > >>> >> >> >> >> virag, if your oozie servers dont have sharelibs > > currently > > > >>> >> >>installed > > > >>> >> >> >> >>then > > > >>> >> >> >> >> unstalling sharelibs as part of an update would not > break > > > ant > > > >>> >> >>running > > > >>> >> >> >> >>app. > > > >>> >> >> >> >> regarding the requirement of enabling the sharelibs to > > have > > > >>> the > > > >>> >> >> >>mains, > > > >>> >> >> >> >>you > > > >>> >> >> >> >> have a point. we need to figure out how to get those in > > the > > > >>> >> >> >>classpath if > > > >>> >> >> >> >> the user is not using the sharelibs. i'd suggest we > leave > > > >>> this > > > >>> >> >> >> >> patch in trunk for now and we move it to a release > branch > > > >>> once we > > > >>> >> >> >>sorted > > > >>> >> >> >> >> out a way of solving this incompatibility. > > > >>> >> >> >> >> > > > >>> >> >> >> >> thanks > > > >>> >> >> >> >> > > > >>> >> >> >> >> Alejandro > > > >>> >> >> >> >> (phone typing) > > > >>> >> >> >> >> > > > >>> >> >> >> >> On Apr 23, 2013, at 5:09 PM, Virag Kothari < > > > >>> vi...@yahoo-inc.com> > > > >>> >> >> >>wrote: > > > >>> >> >> >> >> > > > >>> >> >> >> >> > > > > >>> >> >> >> >> > Tucu, Thanks for the reply and explanation. > > > >>> >> >> >> >> > For workflows running in Y!, cold upgrade is not a > > > feasible > > > >>> >> >>option > > > >>> >> >> >>as > > > >>> >> >> >> >>we > > > >>> >> >> >> >> > cannot expect all the running actions to complete > > before > > > >>> Oozie > > > >>> >> >> >> >>restart. > > > >>> >> >> >> >> > Hot upgrade is a good option, but we will need time > to > > > >>> >> >>implement it > > > >>> >> >> >> >>as we > > > >>> >> >> >> >> > currently don't have sharelib as part of our > > deployment. > > > >>> >> >> >> >> > So can we deprecate the refactoring of launcher > classes > > > in > > > >>> this > > > >>> >> >> >> >>release? > > > >>> >> >> >> >> > Also, all the launcher classes are only added to > > > >>> distributed > > > >>> >> >>cache > > > >>> >> >> >>if > > > >>> >> >> >> >>the > > > >>> >> >> >> >> > workflow is configured to use system lib path > > > >>> >> >> >> >> > (oozie.use.system.libpath=true) > > > >>> >> >> >> >> > We shouldn't mandate the workflow to have this > property > > > as > > > >>> they > > > >>> >> >>are > > > >>> >> >> >> >> > oozie's launcher classes. > > > >>> >> >> >> >> > > > > >>> >> >> >> >> > Thanks, > > > >>> >> >> >> >> > Virag > > > >>> >> >> >> >> > > > > >>> >> >> >> >> > > > > >>> >> >> >> >> > > > > >>> >> >> >> >> > On 4/23/13 3:21 PM, "Alejandro Abdelnur" < > > > tuc...@gmail.com > > > >>> > > > > >>> >> >>wrote: > > > >>> >> >> >> >> > > > > >>> >> >> >> >> >> Virag, > > > >>> >> >> >> >> >> > > > >>> >> >> >> >> >> On sharelib being required, yes you are correct. For > > > this > > > >>> we > > > >>> >> >> >>should: > > > >>> >> >> >> >> >> > > > >>> >> >> >> >> >> * Make 100% clear in the quick-start/install docs > that > > > the > > > >>> >> >> >>sharelib > > > >>> >> >> >> >>is > > > >>> >> >> >> >> >> REQUIRED. > > > >>> >> >> >> >> >> > > > >>> >> >> >> >> >> * Add a check at Oozie startup to verify the > sharelib > > > dir > > > >>> >> >>exists, > > > >>> >> >> >> >>else > > > >>> >> >> >> >> >> fail > > > >>> >> >> >> >> >> to start. > > > >>> >> >> >> >> >> > > > >>> >> >> >> >> >> On sharelib lib issue during upgrade for in-flight > > jobs. > > > >>> >> >> >>Depending on > > > >>> >> >> >> >> the > > > >>> >> >> >> >> >> type of upgrade this may be an issue. > > > >>> >> >> >> >> >> > > > >>> >> >> >> >> >> * If you are upgrading an oozie server fix that does > > not > > > >>> >> change > > > >>> >> >> >>the > > > >>> >> >> >> >> >> sharelib files, this is not an issue and you can > just > > > >>> shutdown > > > >>> >> >>the > > > >>> >> >> >> >>oozie > > > >>> >> >> >> >> >> server with in-flight jobs. > > > >>> >> >> >> >> >> > > > >>> >> >> >> >> >> * If you are upgrading an oozie server fix that > > involves > > > >>> >> >>sharelib > > > >>> >> >> >> >>files > > > >>> >> >> >> >> >> changes, then you have 2 options: > > > >>> >> >> >> >> >> > > > >>> >> >> >> >> >> ** Cold upgrade: bring all WF jobs to > > > suspend/completion, > > > >>> wait > > > >>> >> >> >>till > > > >>> >> >> >> >>all > > > >>> >> >> >> >> >> running actions end, then shutdown oozie server, > > upgrade > > > >>> oozie > > > >>> >> >> >>server > > > >>> >> >> >> >> and > > > >>> >> >> >> >> >> sharelib. Then restart oozie server and resume WF > > jobs. > > > In > > > >>> >> this > > > >>> >> >> >>case > > > >>> >> >> >> >>all > > > >>> >> >> >> >> >> new WF actions will use the new sharelib. > > > >>> >> >> >> >> >> > > > >>> >> >> >> >> >> ** Hot upgrade: stop oozie server. modify the > > > >>> oozie-site.xml > > > >>> >> >> >>sharelib > > > >>> >> >> >> >> >> location to point to a new directory. upgrade the > > oozie > > > >>> >> server. > > > >>> >> >> >> >>install > > > >>> >> >> >> >> >> the > > > >>> >> >> >> >> >> sharelib (will be a create as the sharelib dir in > HDFS > > > >>> does > > > >>> >> not > > > >>> >> >> >> >>exist). > > > >>> >> >> >> >> >> start the oozie server. In this case all running WF > > > >>> actions > > > >>> >> >>will > > > >>> >> >> >> >> continue > > > >>> >> >> >> >> >> running with no issues as the JARs in the > distributed > > > >>> cache > > > >>> >> >>have > > > >>> >> >> >>not > > > >>> >> >> >> >> been > > > >>> >> >> >> >> >> touch. All new WF actions will start using the new > > > >>> sharelib. > > > >>> >> >> >> >> >> > > > >>> >> >> >> >> >> Note that this sharelib upgrade protocol is not > > > >>> introduced by > > > >>> >> >> >> >>requiring > > > >>> >> >> >> >> >> sharelib, it is required if you have applications > that > > > use > > > >>> >> >> >>sharelib > > > >>> >> >> >> >> today. > > > >>> >> >> >> >> >> > > > >>> >> >> >> >> >> Does this address your concerns? > > > >>> >> >> >> >> >> > > > >>> >> >> >> >> >> Thanks. > > > >>> >> >> >> >> >> > > > >>> >> >> >> >> >> > > > >>> >> >> >> >> >> > > > >>> >> >> >> >> >> On Tue, Apr 23, 2013 at 1:35 PM, Virag Kothari > > > >>> >> >> >><vi...@yahoo-inc.com> > > > >>> >> >> >> >> >> wrote: > > > >>> >> >> >> >> >> > > > >>> >> >> >> >> >>> Hi, > > > >>> >> >> >> >> >>> > > > >>> >> >> >> >> >>> With OOZIE-1311 and its subtasks, the idea seems > to > > > >>> move all > > > >>> >> >>the > > > >>> >> >> >> >> >>> launcher > > > >>> >> >> >> >> >>> classes like PigMain, HiveMain etc. to their > > > respective > > > >>> >> >> >>sharelibs. > > > >>> >> >> >> >> >>> So, now shared lib is a mandatory deployment step. > > > Before > > > >>> >> >>shared > > > >>> >> >> >>lib > > > >>> >> >> >> >> was > > > >>> >> >> >> >> >>> optional as users could bundle jars with their > > workflow > > > >>> >> >> >>application. > > > >>> >> >> >> >> >>> So always requiring shared lib seems to introduce 2 > > > >>> problems: > > > >>> >> >> >> >> >>> > > > >>> >> >> >> >> >>> 1. The current deployments which don't use action > > > >>> shared > > > >>> >> lib > > > >>> >> >> >>will > > > >>> >> >> >> >> >>> fail. > > > >>> >> >> >> >> >>> So, probably we should deprecate the current > > behavior. > > > >>> >> >> >> >> >>> > > > >>> >> >> >> >> >>> 2. The hadoop distributed cache mechanism will > fail a > > > >>> job if > > > >>> >> >>the > > > >>> >> >> >> >>files > > > >>> >> >> >> >> >>> in > > > >>> >> >> >> >> >>> DC are updated on hdfs while the hadoop job is > > running. > > > >>> So, > > > >>> >> >>when > > > >>> >> >> >> >>Oozie > > > >>> >> >> >> >> >>> is > > > >>> >> >> >> >> >>> restarted and shared lib is uploaded to hdfs as > part > > of > > > >>> >> >> >> >> >>> deployment, hadoop will fail the > > existing > > > >>> jobs > > > >>> >> >>for > > > >>> >> >> >> >>which > > > >>> >> >> >> >> >>> the > > > >>> >> >> >> >> >>> timestamp of the file on hdfs doesn't match the > > > >>> timestamp of > > > >>> >> >>its > > > >>> >> >> >> >>copy > > > >>> >> >> >> >> >>> in > > > >>> >> >> >> >> >>> the job's DC. > > > >>> >> >> >> >> >>> > > > >>> >> >> >> >> >>> > > > >>> >> >> >> >> >>> Thanks, > > > >>> >> >> >> >> >>> Virag > > > >>> >> >> >> >> > > > > >>> >> >> >> >> > > > >>> >> >> >> > > > >>> >> >> >> > > > >>> >> >> > > > >>> >> >> > > > >>> >> > > > >>> >> > > > >>> > > > > >>> > > > >> > > > >> > > > > > > > > > > > > -- > > > > Alejandro > > > > > > > > > >
