It took forever, but I finally got the refactoring done; I just uploaded a patch at OOZIE-1315 <https://issues.apache.org/jira/browse/OOZIE-1315>. Please take a look.
The patch has the default set to true (use the launcher jar). When I commit it I can have the version in branch-4 use true and the version in trunk use false. thanks - Robert On Mon, May 6, 2013 at 3:06 PM, Rohini Palaniswamy <rohini.adi...@gmail.com>wrote: > Tucu, > That sounds good. Can we have the changes go in trunk first and once we > validate that everything is fine, can we then port all the patches to > branch-4.0? > > Thanks, > Rohini > > > On Mon, May 6, 2013 at 2:52 PM, Alejandro Abdelnur <t...@cloudera.com > >wrote: > > > Rohini, > > > > I understand you don't want to destabilize Oozie 4. Still, as we have no > > done a 4 release yet, I think it is a good time to introduce this > feature. > > > > How about the following (similar to what we did for command > interruptions)? > > > > We do as Robert proposed in his last email, but keeping the default value > > on OFF for Oozie 4; and in Oozie 5 we invert the default value to ON. > > > > This would mean you don't have to change anything in your deployment > > protocol for Oozie 4. > > > > Thx > > > > > > > > On Mon, May 6, 2013 at 2:30 PM, Rohini Palaniswamy < > > rohini.adi...@gmail.com> wrote: > > > >> Robert, > >> > A new major release, like Oozie 4, would be a good time to introduce > >> this change as the default; > >> > >> We already have Oozie 4 deployed in some clusters and it is stable > >> now. We are waiting for users to validate Oozie-HCat integration before > we > >> deploy them in production clusters and call out a release in Apache. So > it > >> would be good to introduce this in the next release which will also be a > >> big release anyway. > >> > >> Regards, > >> Rohini > >> > >> > >> On Mon, May 6, 2013 at 2:09 PM, Robert Kanter <rkan...@cloudera.com > >wrote: > >> > >>> Sorry, I realize that I wasn't clear on that point. The Main classes > >>> would > >>> go in their respective sharelib (e.g. PigMain goes in share/lib/pig/) > but > >>> the other few classes that all actions use in the launcher jar (e.g. > >>> LauncherMapper) would go in the oozie sharelib (share/lib/oozie), which > >>> IIRC is already included when you use sharelibs (for example, when you > >>> use > >>> the sharelib with the Pig action, it includes both the share/lib/pig > and > >>> share/lib/oozie). > >>> > >>> thanks > >>> - Robert > >>> > >>> > >>> On Mon, May 6, 2013 at 1:52 PM, Virag Kothari <vi...@yahoo-inc.com> > >>> wrote: > >>> > >>> > Hi Robert, > >>> > > >>> > I didn't understand completely. Are you suggesting to have all > action > >>> > specific main classes in /share/lib/oozie instead of their respective > >>> > action sharelibs? > >>> > > >>> > Thanks, > >>> > Virag > >>> > > >>> > From: Robert Kanter <rkan...@cloudera.com> > >>> > Date: Monday, May 6, 2013 12:47 PM > >>> > To: "dev@oozie.apache.org" <dev@oozie.apache.org>, Virag Kothari < > >>> > vi...@yahoo-inc.com>, Alejandro Abdelnur <t...@cloudera.com> > >>> > Subject: Re: OOZIE-1311 > >>> > > >>> > Virag, > >>> > I talked more with Alejandro about the sharelib and the launcher jar. > >>> > > >>> > It would be good if we could eventually get rid of the launcher jar > >>> > completely because: > >>> > - less files would be copied to HDFS so it should be faster > >>> > - it would fix an issue where some linux distress cleanup the temp > dir > >>> and > >>> > delete the launcher jar > >>> > > >>> > The contents of the launcher jar could be moved to the Oozie > sharelib > >>> > (/share/lib/oozie) which IIRC currently only has a json jar file. > >>> > > >>> > As for backwards compatibility, we could do something like what you > >>> > suggested for the other sharelibs where the webapp module includes > >>> them. > >>> > Then we'd add some property to oozie-site that would switch between > >>> the > >>> > old behavior with the launcher jar (sharelib is not required) and the > >>> new > >>> > behavior without the launcher jar (sharelib is required). A new > major > >>> > release, like Oozie 4, would be a good time to introduce this change > >>> as the > >>> > default; users can use the property to go back to the old behavior. > >>> Also, > >>> > if set to use the new behavior, we should make > >>> oozie.use.system.libpath in > >>> > the job.properties default to true because the sharelib would be > >>> required, > >>> > and if set to the old behavior, it would default to false. > >>> > > >>> > Thoughts? > >>> > > >>> > thanks > >>> > - Robert > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > On Wed, Apr 24, 2013 at 12:24 PM, Virag Kothari <vi...@yahoo-inc.com > >>> >wrote: > >>> > > >>> >> Thanks Robert. Created OOZIE-1341 > >>> >> > >>> >> On 4/24/13 11:36 AM, "Robert Kanter" <rkan...@cloudera.com> wrote: > >>> >> > >>> >> >I think I follow what you're saying; assuming everything works > >>> correctly, > >>> >> >I > >>> >> >think that sounds good. Can you make a JIRA with that and put up a > >>> >> patch; > >>> >> >that might be easier to see how this would work. > >>> >> > > >>> >> >thanks > >>> >> >- Robert > >>> >> > > >>> >> > > >>> >> >On Wed, Apr 24, 2013 at 11:29 AM, Virag Kothari < > vi...@yahoo-inc.com > >>> > > >>> >> >wrote: > >>> >> > > >>> >> >> We can keep the Main classes in the share lib, but have the > action > >>> >> >> executors package the main classes for shipping it to > >>> >> >> Launcher (basically again having the method getLauncherClasses() > in > >>> >> >>action > >>> >> >> executors). Also the oozie-sharelib.jar containing main classes > >>> >> >> should be part of web-app. > >>> >> >> This will improve the building/testing env as classes are in > >>> isolated > >>> >> >> sharelibs but will make the share lib optional as main classes > are > >>> part > >>> >> >>of > >>> >> >> oozie-server. > >>> >> >> Can we do this change to trunk as this is blocking our QE's to > pick > >>> >> >>builds > >>> >> >> due to share lib incompatibility? > >>> >> >> > >>> >> >> We can still do what oozie-1318 is trying to achieve. For, e.g > with > >>> >> Hcat > >>> >> >> integration, > >>> >> >> there can be multiple overridable implementation of URIHandlers > >>> that > >>> >> can > >>> >> >> be shipped to launcher. > >>> >> >> We can do the same for other main classes if required. > >>> >> >> > >>> >> >> > >>> >> >> Thanks, > >>> >> >> Virag > >>> >> >> > >>> >> >> > >>> >> >> On 4/24/13 10:18 AM, "Robert Kanter" <rkan...@cloudera.com> > wrote: > >>> >> >> > >>> >> >> >That's true, its primarily a building/testing improvement. And > >>> from a > >>> >> >> >design perspective, it could be considered "cleaner" to not have > >>> the > >>> >> >> >launcher jar. > >>> >> >> > > >>> >> >> >Your idea sounds like a good compromise between backwards > >>> >> compatibility > >>> >> >> >and > >>> >> >> >improving the building/testing env. How would we do that? > >>> >> >> > > >>> >> >> >As a side note, there's also a new feature with the Main classes > >>> being > >>> >> >>in > >>> >> >> >the sharelib; users can now override the Main class that Oozie > >>> will > >>> >> use > >>> >> >> >for > >>> >> >> >the action because its no longer in the launcher jar. I think > >>> this > >>> >> >>could > >>> >> >> >still be a useful feature, but I'm guessing that if we do the > >>> >> >>compromise, > >>> >> >> >this would go away? In any case, OOZIE-1318 is to make the Main > >>> >> >> >overridable with a config, which is probably cleaner and more > >>> flexible > >>> >> >> >than > >>> >> >> >replacing a jar file. > >>> >> >> > > >>> >> >> >thanks > >>> >> >> >- Robert > >>> >> >> > > >>> >> >> > > >>> >> >> >On Wed, Apr 24, 2013 at 9:54 AM, Virag Kothari < > >>> vi...@yahoo-inc.com> > >>> >> >> >wrote: > >>> >> >> > > >>> >> >> >> That is a good suggestion for preserving compatibility, > Robert. > >>> >> >> >> > >>> >> >> >> Another question is what is the main purpose of OOZIE-1311? > From > >>> >> JIRA > >>> >> >> >> description, the primary > >>> >> >> >> advantage of moving Main classes from core to share lib is it > >>> can > >>> >> >>help > >>> >> >> >>in > >>> >> >> >> preventing dependencies conflicts > >>> >> >> >> (different versions of antlr's for pig, hive). However, that > >>> problem > >>> >> >> >>only > >>> >> >> >> exists while building/testing Oozie. > >>> >> >> >> If that is the case, main classes can be moved to share lib > but > >>> they > >>> >> >>can > >>> >> >> >> be bundled in oozie-server itself. > >>> >> >> >> In that case, the deployment doesn't change and share lib can > >>> still > >>> >> >>be > >>> >> >> >> optional > >>> >> >> >> as oozie server would ship those classes (same as what was > >>> happening > >>> >> >> >> before). And, > >>> >> >> >> advantage of 1311 for building/testing will be retained > >>> >> >> >> > >>> >> >> >> The main concern is having share lib as a required > installation > >>> will > >>> >> >> >> complicate deployment. > >>> >> >> >> Doing hot upgrade for share lib is non-trivial and Oozie > doesn't > >>> >> >>provide > >>> >> >> >> inbuilt support for that. > >>> >> >> >> Even though end-users wouldn't notice the change, it will be a > >>> big > >>> >> >> >>change > >>> >> >> >> for system admins installing oozie > >>> >> >> >> So even though share lib is nice feature to have, keeping it > >>> >> optional > >>> >> >> >> seems better to me as there are other ways > >>> >> >> >> Of including jars and which were introduced much before share > >>> lib. > >>> >> >> >> > >>> >> >> >> Thanks, > >>> >> >> >> Virag > >>> >> >> >> > >>> >> >> >> > >>> >> >> >> > >>> >> >> >> > >>> >> >> >> On 4/24/13 9:09 AM, "Robert Kanter" <rkan...@cloudera.com> > >>> wrote: > >>> >> >> >> > >>> >> >> >> >Alejandro had another suggestion to prevent breaking stuff > for > >>> >> >>existing > >>> >> >> >> >apps where the sharelib isn't used. Assuming the sharelib is > >>> >> >> >>installed, > >>> >> >> >> >if > >>> >> >> >> >the user doesn't set oozie.use.system.libpath to true, then > >>> Oozie > >>> >> >>would > >>> >> >> >> >only include the oozie-*.jar from the sharelib (the one with > >>> the > >>> >> >>Main) > >>> >> >> >>in > >>> >> >> >> >it; if its set to true, then Oozie would include all of the > >>> jars in > >>> >> >>the > >>> >> >> >> >sharelib. This would allow users to still use their own lib > >>> dir > >>> >> >>while > >>> >> >> >> >still getting the Main into the classpath. > >>> >> >> >> > > >>> >> >> >> >- Robert > >>> >> >> >> > > >>> >> >> >> > > >>> >> >> >> > > >>> >> >> >> >On Tue, Apr 23, 2013 at 10:46 PM, Alejandro Abdelnur > >>> >> >> >> ><tuc...@gmail.com>wrote: > >>> >> >> >> > > >>> >> >> >> >> virag, if your oozie servers dont have sharelibs currently > >>> >> >>installed > >>> >> >> >> >>then > >>> >> >> >> >> unstalling sharelibs as part of an update would not break > ant > >>> >> >>running > >>> >> >> >> >>app. > >>> >> >> >> >> regarding the requirement of enabling the sharelibs to have > >>> the > >>> >> >> >>mains, > >>> >> >> >> >>you > >>> >> >> >> >> have a point. we need to figure out how to get those in the > >>> >> >> >>classpath if > >>> >> >> >> >> the user is not using the sharelibs. i'd suggest we leave > >>> this > >>> >> >> >> >> patch in trunk for now and we move it to a release branch > >>> once we > >>> >> >> >>sorted > >>> >> >> >> >> out a way of solving this incompatibility. > >>> >> >> >> >> > >>> >> >> >> >> thanks > >>> >> >> >> >> > >>> >> >> >> >> Alejandro > >>> >> >> >> >> (phone typing) > >>> >> >> >> >> > >>> >> >> >> >> On Apr 23, 2013, at 5:09 PM, Virag Kothari < > >>> vi...@yahoo-inc.com> > >>> >> >> >>wrote: > >>> >> >> >> >> > >>> >> >> >> >> > > >>> >> >> >> >> > Tucu, Thanks for the reply and explanation. > >>> >> >> >> >> > For workflows running in Y!, cold upgrade is not a > feasible > >>> >> >>option > >>> >> >> >>as > >>> >> >> >> >>we > >>> >> >> >> >> > cannot expect all the running actions to complete before > >>> Oozie > >>> >> >> >> >>restart. > >>> >> >> >> >> > Hot upgrade is a good option, but we will need time to > >>> >> >>implement it > >>> >> >> >> >>as we > >>> >> >> >> >> > currently don't have sharelib as part of our deployment. > >>> >> >> >> >> > So can we deprecate the refactoring of launcher classes > in > >>> this > >>> >> >> >> >>release? > >>> >> >> >> >> > Also, all the launcher classes are only added to > >>> distributed > >>> >> >>cache > >>> >> >> >>if > >>> >> >> >> >>the > >>> >> >> >> >> > workflow is configured to use system lib path > >>> >> >> >> >> > (oozie.use.system.libpath=true) > >>> >> >> >> >> > We shouldn't mandate the workflow to have this property > as > >>> they > >>> >> >>are > >>> >> >> >> >> > oozie's launcher classes. > >>> >> >> >> >> > > >>> >> >> >> >> > Thanks, > >>> >> >> >> >> > Virag > >>> >> >> >> >> > > >>> >> >> >> >> > > >>> >> >> >> >> > > >>> >> >> >> >> > On 4/23/13 3:21 PM, "Alejandro Abdelnur" < > tuc...@gmail.com > >>> > > >>> >> >>wrote: > >>> >> >> >> >> > > >>> >> >> >> >> >> Virag, > >>> >> >> >> >> >> > >>> >> >> >> >> >> On sharelib being required, yes you are correct. For > this > >>> we > >>> >> >> >>should: > >>> >> >> >> >> >> > >>> >> >> >> >> >> * Make 100% clear in the quick-start/install docs that > the > >>> >> >> >>sharelib > >>> >> >> >> >>is > >>> >> >> >> >> >> REQUIRED. > >>> >> >> >> >> >> > >>> >> >> >> >> >> * Add a check at Oozie startup to verify the sharelib > dir > >>> >> >>exists, > >>> >> >> >> >>else > >>> >> >> >> >> >> fail > >>> >> >> >> >> >> to start. > >>> >> >> >> >> >> > >>> >> >> >> >> >> On sharelib lib issue during upgrade for in-flight jobs. > >>> >> >> >>Depending on > >>> >> >> >> >> the > >>> >> >> >> >> >> type of upgrade this may be an issue. > >>> >> >> >> >> >> > >>> >> >> >> >> >> * If you are upgrading an oozie server fix that does not > >>> >> change > >>> >> >> >>the > >>> >> >> >> >> >> sharelib files, this is not an issue and you can just > >>> shutdown > >>> >> >>the > >>> >> >> >> >>oozie > >>> >> >> >> >> >> server with in-flight jobs. > >>> >> >> >> >> >> > >>> >> >> >> >> >> * If you are upgrading an oozie server fix that involves > >>> >> >>sharelib > >>> >> >> >> >>files > >>> >> >> >> >> >> changes, then you have 2 options: > >>> >> >> >> >> >> > >>> >> >> >> >> >> ** Cold upgrade: bring all WF jobs to > suspend/completion, > >>> wait > >>> >> >> >>till > >>> >> >> >> >>all > >>> >> >> >> >> >> running actions end, then shutdown oozie server, upgrade > >>> oozie > >>> >> >> >>server > >>> >> >> >> >> and > >>> >> >> >> >> >> sharelib. Then restart oozie server and resume WF jobs. > In > >>> >> this > >>> >> >> >>case > >>> >> >> >> >>all > >>> >> >> >> >> >> new WF actions will use the new sharelib. > >>> >> >> >> >> >> > >>> >> >> >> >> >> ** Hot upgrade: stop oozie server. modify the > >>> oozie-site.xml > >>> >> >> >>sharelib > >>> >> >> >> >> >> location to point to a new directory. upgrade the oozie > >>> >> server. > >>> >> >> >> >>install > >>> >> >> >> >> >> the > >>> >> >> >> >> >> sharelib (will be a create as the sharelib dir in HDFS > >>> does > >>> >> not > >>> >> >> >> >>exist). > >>> >> >> >> >> >> start the oozie server. In this case all running WF > >>> actions > >>> >> >>will > >>> >> >> >> >> continue > >>> >> >> >> >> >> running with no issues as the JARs in the distributed > >>> cache > >>> >> >>have > >>> >> >> >>not > >>> >> >> >> >> been > >>> >> >> >> >> >> touch. All new WF actions will start using the new > >>> sharelib. > >>> >> >> >> >> >> > >>> >> >> >> >> >> Note that this sharelib upgrade protocol is not > >>> introduced by > >>> >> >> >> >>requiring > >>> >> >> >> >> >> sharelib, it is required if you have applications that > use > >>> >> >> >>sharelib > >>> >> >> >> >> today. > >>> >> >> >> >> >> > >>> >> >> >> >> >> Does this address your concerns? > >>> >> >> >> >> >> > >>> >> >> >> >> >> Thanks. > >>> >> >> >> >> >> > >>> >> >> >> >> >> > >>> >> >> >> >> >> > >>> >> >> >> >> >> On Tue, Apr 23, 2013 at 1:35 PM, Virag Kothari > >>> >> >> >><vi...@yahoo-inc.com> > >>> >> >> >> >> >> wrote: > >>> >> >> >> >> >> > >>> >> >> >> >> >>> Hi, > >>> >> >> >> >> >>> > >>> >> >> >> >> >>> With OOZIE-1311 and its subtasks, the idea seems to > >>> move all > >>> >> >>the > >>> >> >> >> >> >>> launcher > >>> >> >> >> >> >>> classes like PigMain, HiveMain etc. to their > respective > >>> >> >> >>sharelibs. > >>> >> >> >> >> >>> So, now shared lib is a mandatory deployment step. > Before > >>> >> >>shared > >>> >> >> >>lib > >>> >> >> >> >> was > >>> >> >> >> >> >>> optional as users could bundle jars with their workflow > >>> >> >> >>application. > >>> >> >> >> >> >>> So always requiring shared lib seems to introduce 2 > >>> problems: > >>> >> >> >> >> >>> > >>> >> >> >> >> >>> 1. The current deployments which don't use action > >>> shared > >>> >> lib > >>> >> >> >>will > >>> >> >> >> >> >>> fail. > >>> >> >> >> >> >>> So, probably we should deprecate the current behavior. > >>> >> >> >> >> >>> > >>> >> >> >> >> >>> 2. The hadoop distributed cache mechanism will fail a > >>> job if > >>> >> >>the > >>> >> >> >> >>files > >>> >> >> >> >> >>> in > >>> >> >> >> >> >>> DC are updated on hdfs while the hadoop job is running. > >>> So, > >>> >> >>when > >>> >> >> >> >>Oozie > >>> >> >> >> >> >>> is > >>> >> >> >> >> >>> restarted and shared lib is uploaded to hdfs as part of > >>> >> >> >> >> >>> deployment, hadoop will fail the existing > >>> jobs > >>> >> >>for > >>> >> >> >> >>which > >>> >> >> >> >> >>> the > >>> >> >> >> >> >>> timestamp of the file on hdfs doesn't match the > >>> timestamp of > >>> >> >>its > >>> >> >> >> >>copy > >>> >> >> >> >> >>> in > >>> >> >> >> >> >>> the job's DC. > >>> >> >> >> >> >>> > >>> >> >> >> >> >>> > >>> >> >> >> >> >>> Thanks, > >>> >> >> >> >> >>> Virag > >>> >> >> >> >> > > >>> >> >> >> >> > >>> >> >> >> > >>> >> >> >> > >>> >> >> > >>> >> >> > >>> >> > >>> >> > >>> > > >>> > >> > >> > > > > > > -- > > Alejandro > > >
