Robert, Do you want to spend effort to rebase and put this patch in branch-4 also? I was wondering since we were thinking of releasing trunk with SLA work along with HCat as 4.0 instead of releasing the current branch-4 to consolidate the database changes. If you would like to release branch-4, then we need to do the OozieDBCLI table changes for SLA work as 5.0 and change trunk to 5.0 then.
Regards, Rohini On Thu, Jun 6, 2013 at 10:04 AM, Robert Kanter <rkan...@cloudera.com> wrote: > It took forever, but I finally got the refactoring done; I just uploaded a > patch at OOZIE-1315 <https://issues.apache.org/jira/browse/OOZIE-1315>. > Please take a look. > > The patch has the default set to true (use the launcher jar). When I > commit it I can have the version in branch-4 use true and the version in > trunk use false. > > thanks > - Robert > > > On Mon, May 6, 2013 at 3:06 PM, Rohini Palaniswamy > <rohini.adi...@gmail.com>wrote: > > > Tucu, > > That sounds good. Can we have the changes go in trunk first and once > we > > validate that everything is fine, can we then port all the patches to > > branch-4.0? > > > > Thanks, > > Rohini > > > > > > On Mon, May 6, 2013 at 2:52 PM, Alejandro Abdelnur <t...@cloudera.com > > >wrote: > > > > > Rohini, > > > > > > I understand you don't want to destabilize Oozie 4. Still, as we have > no > > > done a 4 release yet, I think it is a good time to introduce this > > feature. > > > > > > How about the following (similar to what we did for command > > interruptions)? > > > > > > We do as Robert proposed in his last email, but keeping the default > value > > > on OFF for Oozie 4; and in Oozie 5 we invert the default value to ON. > > > > > > This would mean you don't have to change anything in your deployment > > > protocol for Oozie 4. > > > > > > Thx > > > > > > > > > > > > On Mon, May 6, 2013 at 2:30 PM, Rohini Palaniswamy < > > > rohini.adi...@gmail.com> wrote: > > > > > >> Robert, > > >> > A new major release, like Oozie 4, would be a good time to introduce > > >> this change as the default; > > >> > > >> We already have Oozie 4 deployed in some clusters and it is stable > > >> now. We are waiting for users to validate Oozie-HCat integration > before > > we > > >> deploy them in production clusters and call out a release in Apache. > So > > it > > >> would be good to introduce this in the next release which will also > be a > > >> big release anyway. > > >> > > >> Regards, > > >> Rohini > > >> > > >> > > >> On Mon, May 6, 2013 at 2:09 PM, Robert Kanter <rkan...@cloudera.com > > >wrote: > > >> > > >>> Sorry, I realize that I wasn't clear on that point. The Main classes > > >>> would > > >>> go in their respective sharelib (e.g. PigMain goes in share/lib/pig/) > > but > > >>> the other few classes that all actions use in the launcher jar (e.g. > > >>> LauncherMapper) would go in the oozie sharelib (share/lib/oozie), > which > > >>> IIRC is already included when you use sharelibs (for example, when > you > > >>> use > > >>> the sharelib with the Pig action, it includes both the share/lib/pig > > and > > >>> share/lib/oozie). > > >>> > > >>> thanks > > >>> - Robert > > >>> > > >>> > > >>> On Mon, May 6, 2013 at 1:52 PM, Virag Kothari <vi...@yahoo-inc.com> > > >>> wrote: > > >>> > > >>> > Hi Robert, > > >>> > > > >>> > I didn't understand completely. Are you suggesting to have all > > action > > >>> > specific main classes in /share/lib/oozie instead of their > respective > > >>> > action sharelibs? > > >>> > > > >>> > Thanks, > > >>> > Virag > > >>> > > > >>> > From: Robert Kanter <rkan...@cloudera.com> > > >>> > Date: Monday, May 6, 2013 12:47 PM > > >>> > To: "dev@oozie.apache.org" <dev@oozie.apache.org>, Virag Kothari < > > >>> > vi...@yahoo-inc.com>, Alejandro Abdelnur <t...@cloudera.com> > > >>> > Subject: Re: OOZIE-1311 > > >>> > > > >>> > Virag, > > >>> > I talked more with Alejandro about the sharelib and the launcher > jar. > > >>> > > > >>> > It would be good if we could eventually get rid of the launcher jar > > >>> > completely because: > > >>> > - less files would be copied to HDFS so it should be faster > > >>> > - it would fix an issue where some linux distress cleanup the temp > > dir > > >>> and > > >>> > delete the launcher jar > > >>> > > > >>> > The contents of the launcher jar could be moved to the Oozie > > sharelib > > >>> > (/share/lib/oozie) which IIRC currently only has a json jar file. > > >>> > > > >>> > As for backwards compatibility, we could do something like what > you > > >>> > suggested for the other sharelibs where the webapp module includes > > >>> them. > > >>> > Then we'd add some property to oozie-site that would switch > between > > >>> the > > >>> > old behavior with the launcher jar (sharelib is not required) and > the > > >>> new > > >>> > behavior without the launcher jar (sharelib is required). A new > > major > > >>> > release, like Oozie 4, would be a good time to introduce this > change > > >>> as the > > >>> > default; users can use the property to go back to the old behavior. > > >>> Also, > > >>> > if set to use the new behavior, we should make > > >>> oozie.use.system.libpath in > > >>> > the job.properties default to true because the sharelib would be > > >>> required, > > >>> > and if set to the old behavior, it would default to false. > > >>> > > > >>> > Thoughts? > > >>> > > > >>> > thanks > > >>> > - Robert > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > On Wed, Apr 24, 2013 at 12:24 PM, Virag Kothari < > vi...@yahoo-inc.com > > >>> >wrote: > > >>> > > > >>> >> Thanks Robert. Created OOZIE-1341 > > >>> >> > > >>> >> On 4/24/13 11:36 AM, "Robert Kanter" <rkan...@cloudera.com> > wrote: > > >>> >> > > >>> >> >I think I follow what you're saying; assuming everything works > > >>> correctly, > > >>> >> >I > > >>> >> >think that sounds good. Can you make a JIRA with that and put > up a > > >>> >> patch; > > >>> >> >that might be easier to see how this would work. > > >>> >> > > > >>> >> >thanks > > >>> >> >- Robert > > >>> >> > > > >>> >> > > > >>> >> >On Wed, Apr 24, 2013 at 11:29 AM, Virag Kothari < > > vi...@yahoo-inc.com > > >>> > > > >>> >> >wrote: > > >>> >> > > > >>> >> >> We can keep the Main classes in the share lib, but have the > > action > > >>> >> >> executors package the main classes for shipping it to > > >>> >> >> Launcher (basically again having the method > getLauncherClasses() > > in > > >>> >> >>action > > >>> >> >> executors). Also the oozie-sharelib.jar containing main classes > > >>> >> >> should be part of web-app. > > >>> >> >> This will improve the building/testing env as classes are in > > >>> isolated > > >>> >> >> sharelibs but will make the share lib optional as main classes > > are > > >>> part > > >>> >> >>of > > >>> >> >> oozie-server. > > >>> >> >> Can we do this change to trunk as this is blocking our QE's to > > pick > > >>> >> >>builds > > >>> >> >> due to share lib incompatibility? > > >>> >> >> > > >>> >> >> We can still do what oozie-1318 is trying to achieve. For, e.g > > with > > >>> >> Hcat > > >>> >> >> integration, > > >>> >> >> there can be multiple overridable implementation of URIHandlers > > >>> that > > >>> >> can > > >>> >> >> be shipped to launcher. > > >>> >> >> We can do the same for other main classes if required. > > >>> >> >> > > >>> >> >> > > >>> >> >> Thanks, > > >>> >> >> Virag > > >>> >> >> > > >>> >> >> > > >>> >> >> On 4/24/13 10:18 AM, "Robert Kanter" <rkan...@cloudera.com> > > wrote: > > >>> >> >> > > >>> >> >> >That's true, its primarily a building/testing improvement. > And > > >>> from a > > >>> >> >> >design perspective, it could be considered "cleaner" to not > have > > >>> the > > >>> >> >> >launcher jar. > > >>> >> >> > > > >>> >> >> >Your idea sounds like a good compromise between backwards > > >>> >> compatibility > > >>> >> >> >and > > >>> >> >> >improving the building/testing env. How would we do that? > > >>> >> >> > > > >>> >> >> >As a side note, there's also a new feature with the Main > classes > > >>> being > > >>> >> >>in > > >>> >> >> >the sharelib; users can now override the Main class that Oozie > > >>> will > > >>> >> use > > >>> >> >> >for > > >>> >> >> >the action because its no longer in the launcher jar. I think > > >>> this > > >>> >> >>could > > >>> >> >> >still be a useful feature, but I'm guessing that if we do the > > >>> >> >>compromise, > > >>> >> >> >this would go away? In any case, OOZIE-1318 is to make the > Main > > >>> >> >> >overridable with a config, which is probably cleaner and more > > >>> flexible > > >>> >> >> >than > > >>> >> >> >replacing a jar file. > > >>> >> >> > > > >>> >> >> >thanks > > >>> >> >> >- Robert > > >>> >> >> > > > >>> >> >> > > > >>> >> >> >On Wed, Apr 24, 2013 at 9:54 AM, Virag Kothari < > > >>> vi...@yahoo-inc.com> > > >>> >> >> >wrote: > > >>> >> >> > > > >>> >> >> >> That is a good suggestion for preserving compatibility, > > Robert. > > >>> >> >> >> > > >>> >> >> >> Another question is what is the main purpose of OOZIE-1311? > > From > > >>> >> JIRA > > >>> >> >> >> description, the primary > > >>> >> >> >> advantage of moving Main classes from core to share lib is > it > > >>> can > > >>> >> >>help > > >>> >> >> >>in > > >>> >> >> >> preventing dependencies conflicts > > >>> >> >> >> (different versions of antlr's for pig, hive). However, that > > >>> problem > > >>> >> >> >>only > > >>> >> >> >> exists while building/testing Oozie. > > >>> >> >> >> If that is the case, main classes can be moved to share lib > > but > > >>> they > > >>> >> >>can > > >>> >> >> >> be bundled in oozie-server itself. > > >>> >> >> >> In that case, the deployment doesn't change and share lib > can > > >>> still > > >>> >> >>be > > >>> >> >> >> optional > > >>> >> >> >> as oozie server would ship those classes (same as what was > > >>> happening > > >>> >> >> >> before). And, > > >>> >> >> >> advantage of 1311 for building/testing will be retained > > >>> >> >> >> > > >>> >> >> >> The main concern is having share lib as a required > > installation > > >>> will > > >>> >> >> >> complicate deployment. > > >>> >> >> >> Doing hot upgrade for share lib is non-trivial and Oozie > > doesn't > > >>> >> >>provide > > >>> >> >> >> inbuilt support for that. > > >>> >> >> >> Even though end-users wouldn't notice the change, it will > be a > > >>> big > > >>> >> >> >>change > > >>> >> >> >> for system admins installing oozie > > >>> >> >> >> So even though share lib is nice feature to have, keeping it > > >>> >> optional > > >>> >> >> >> seems better to me as there are other ways > > >>> >> >> >> Of including jars and which were introduced much before > share > > >>> lib. > > >>> >> >> >> > > >>> >> >> >> Thanks, > > >>> >> >> >> Virag > > >>> >> >> >> > > >>> >> >> >> > > >>> >> >> >> > > >>> >> >> >> > > >>> >> >> >> On 4/24/13 9:09 AM, "Robert Kanter" <rkan...@cloudera.com> > > >>> wrote: > > >>> >> >> >> > > >>> >> >> >> >Alejandro had another suggestion to prevent breaking stuff > > for > > >>> >> >>existing > > >>> >> >> >> >apps where the sharelib isn't used. Assuming the sharelib > is > > >>> >> >> >>installed, > > >>> >> >> >> >if > > >>> >> >> >> >the user doesn't set oozie.use.system.libpath to true, then > > >>> Oozie > > >>> >> >>would > > >>> >> >> >> >only include the oozie-*.jar from the sharelib (the one > with > > >>> the > > >>> >> >>Main) > > >>> >> >> >>in > > >>> >> >> >> >it; if its set to true, then Oozie would include all of the > > >>> jars in > > >>> >> >>the > > >>> >> >> >> >sharelib. This would allow users to still use their own > lib > > >>> dir > > >>> >> >>while > > >>> >> >> >> >still getting the Main into the classpath. > > >>> >> >> >> > > > >>> >> >> >> >- Robert > > >>> >> >> >> > > > >>> >> >> >> > > > >>> >> >> >> > > > >>> >> >> >> >On Tue, Apr 23, 2013 at 10:46 PM, Alejandro Abdelnur > > >>> >> >> >> ><tuc...@gmail.com>wrote: > > >>> >> >> >> > > > >>> >> >> >> >> virag, if your oozie servers dont have sharelibs > currently > > >>> >> >>installed > > >>> >> >> >> >>then > > >>> >> >> >> >> unstalling sharelibs as part of an update would not break > > ant > > >>> >> >>running > > >>> >> >> >> >>app. > > >>> >> >> >> >> regarding the requirement of enabling the sharelibs to > have > > >>> the > > >>> >> >> >>mains, > > >>> >> >> >> >>you > > >>> >> >> >> >> have a point. we need to figure out how to get those in > the > > >>> >> >> >>classpath if > > >>> >> >> >> >> the user is not using the sharelibs. i'd suggest we leave > > >>> this > > >>> >> >> >> >> patch in trunk for now and we move it to a release branch > > >>> once we > > >>> >> >> >>sorted > > >>> >> >> >> >> out a way of solving this incompatibility. > > >>> >> >> >> >> > > >>> >> >> >> >> thanks > > >>> >> >> >> >> > > >>> >> >> >> >> Alejandro > > >>> >> >> >> >> (phone typing) > > >>> >> >> >> >> > > >>> >> >> >> >> On Apr 23, 2013, at 5:09 PM, Virag Kothari < > > >>> vi...@yahoo-inc.com> > > >>> >> >> >>wrote: > > >>> >> >> >> >> > > >>> >> >> >> >> > > > >>> >> >> >> >> > Tucu, Thanks for the reply and explanation. > > >>> >> >> >> >> > For workflows running in Y!, cold upgrade is not a > > feasible > > >>> >> >>option > > >>> >> >> >>as > > >>> >> >> >> >>we > > >>> >> >> >> >> > cannot expect all the running actions to complete > before > > >>> Oozie > > >>> >> >> >> >>restart. > > >>> >> >> >> >> > Hot upgrade is a good option, but we will need time to > > >>> >> >>implement it > > >>> >> >> >> >>as we > > >>> >> >> >> >> > currently don't have sharelib as part of our > deployment. > > >>> >> >> >> >> > So can we deprecate the refactoring of launcher classes > > in > > >>> this > > >>> >> >> >> >>release? > > >>> >> >> >> >> > Also, all the launcher classes are only added to > > >>> distributed > > >>> >> >>cache > > >>> >> >> >>if > > >>> >> >> >> >>the > > >>> >> >> >> >> > workflow is configured to use system lib path > > >>> >> >> >> >> > (oozie.use.system.libpath=true) > > >>> >> >> >> >> > We shouldn't mandate the workflow to have this property > > as > > >>> they > > >>> >> >>are > > >>> >> >> >> >> > oozie's launcher classes. > > >>> >> >> >> >> > > > >>> >> >> >> >> > Thanks, > > >>> >> >> >> >> > Virag > > >>> >> >> >> >> > > > >>> >> >> >> >> > > > >>> >> >> >> >> > > > >>> >> >> >> >> > On 4/23/13 3:21 PM, "Alejandro Abdelnur" < > > tuc...@gmail.com > > >>> > > > >>> >> >>wrote: > > >>> >> >> >> >> > > > >>> >> >> >> >> >> Virag, > > >>> >> >> >> >> >> > > >>> >> >> >> >> >> On sharelib being required, yes you are correct. For > > this > > >>> we > > >>> >> >> >>should: > > >>> >> >> >> >> >> > > >>> >> >> >> >> >> * Make 100% clear in the quick-start/install docs that > > the > > >>> >> >> >>sharelib > > >>> >> >> >> >>is > > >>> >> >> >> >> >> REQUIRED. > > >>> >> >> >> >> >> > > >>> >> >> >> >> >> * Add a check at Oozie startup to verify the sharelib > > dir > > >>> >> >>exists, > > >>> >> >> >> >>else > > >>> >> >> >> >> >> fail > > >>> >> >> >> >> >> to start. > > >>> >> >> >> >> >> > > >>> >> >> >> >> >> On sharelib lib issue during upgrade for in-flight > jobs. > > >>> >> >> >>Depending on > > >>> >> >> >> >> the > > >>> >> >> >> >> >> type of upgrade this may be an issue. > > >>> >> >> >> >> >> > > >>> >> >> >> >> >> * If you are upgrading an oozie server fix that does > not > > >>> >> change > > >>> >> >> >>the > > >>> >> >> >> >> >> sharelib files, this is not an issue and you can just > > >>> shutdown > > >>> >> >>the > > >>> >> >> >> >>oozie > > >>> >> >> >> >> >> server with in-flight jobs. > > >>> >> >> >> >> >> > > >>> >> >> >> >> >> * If you are upgrading an oozie server fix that > involves > > >>> >> >>sharelib > > >>> >> >> >> >>files > > >>> >> >> >> >> >> changes, then you have 2 options: > > >>> >> >> >> >> >> > > >>> >> >> >> >> >> ** Cold upgrade: bring all WF jobs to > > suspend/completion, > > >>> wait > > >>> >> >> >>till > > >>> >> >> >> >>all > > >>> >> >> >> >> >> running actions end, then shutdown oozie server, > upgrade > > >>> oozie > > >>> >> >> >>server > > >>> >> >> >> >> and > > >>> >> >> >> >> >> sharelib. Then restart oozie server and resume WF > jobs. > > In > > >>> >> this > > >>> >> >> >>case > > >>> >> >> >> >>all > > >>> >> >> >> >> >> new WF actions will use the new sharelib. > > >>> >> >> >> >> >> > > >>> >> >> >> >> >> ** Hot upgrade: stop oozie server. modify the > > >>> oozie-site.xml > > >>> >> >> >>sharelib > > >>> >> >> >> >> >> location to point to a new directory. upgrade the > oozie > > >>> >> server. > > >>> >> >> >> >>install > > >>> >> >> >> >> >> the > > >>> >> >> >> >> >> sharelib (will be a create as the sharelib dir in HDFS > > >>> does > > >>> >> not > > >>> >> >> >> >>exist). > > >>> >> >> >> >> >> start the oozie server. In this case all running WF > > >>> actions > > >>> >> >>will > > >>> >> >> >> >> continue > > >>> >> >> >> >> >> running with no issues as the JARs in the distributed > > >>> cache > > >>> >> >>have > > >>> >> >> >>not > > >>> >> >> >> >> been > > >>> >> >> >> >> >> touch. All new WF actions will start using the new > > >>> sharelib. > > >>> >> >> >> >> >> > > >>> >> >> >> >> >> Note that this sharelib upgrade protocol is not > > >>> introduced by > > >>> >> >> >> >>requiring > > >>> >> >> >> >> >> sharelib, it is required if you have applications that > > use > > >>> >> >> >>sharelib > > >>> >> >> >> >> today. > > >>> >> >> >> >> >> > > >>> >> >> >> >> >> Does this address your concerns? > > >>> >> >> >> >> >> > > >>> >> >> >> >> >> Thanks. > > >>> >> >> >> >> >> > > >>> >> >> >> >> >> > > >>> >> >> >> >> >> > > >>> >> >> >> >> >> On Tue, Apr 23, 2013 at 1:35 PM, Virag Kothari > > >>> >> >> >><vi...@yahoo-inc.com> > > >>> >> >> >> >> >> wrote: > > >>> >> >> >> >> >> > > >>> >> >> >> >> >>> Hi, > > >>> >> >> >> >> >>> > > >>> >> >> >> >> >>> With OOZIE-1311 and its subtasks, the idea seems to > > >>> move all > > >>> >> >>the > > >>> >> >> >> >> >>> launcher > > >>> >> >> >> >> >>> classes like PigMain, HiveMain etc. to their > > respective > > >>> >> >> >>sharelibs. > > >>> >> >> >> >> >>> So, now shared lib is a mandatory deployment step. > > Before > > >>> >> >>shared > > >>> >> >> >>lib > > >>> >> >> >> >> was > > >>> >> >> >> >> >>> optional as users could bundle jars with their > workflow > > >>> >> >> >>application. > > >>> >> >> >> >> >>> So always requiring shared lib seems to introduce 2 > > >>> problems: > > >>> >> >> >> >> >>> > > >>> >> >> >> >> >>> 1. The current deployments which don't use action > > >>> shared > > >>> >> lib > > >>> >> >> >>will > > >>> >> >> >> >> >>> fail. > > >>> >> >> >> >> >>> So, probably we should deprecate the current > behavior. > > >>> >> >> >> >> >>> > > >>> >> >> >> >> >>> 2. The hadoop distributed cache mechanism will fail a > > >>> job if > > >>> >> >>the > > >>> >> >> >> >>files > > >>> >> >> >> >> >>> in > > >>> >> >> >> >> >>> DC are updated on hdfs while the hadoop job is > running. > > >>> So, > > >>> >> >>when > > >>> >> >> >> >>Oozie > > >>> >> >> >> >> >>> is > > >>> >> >> >> >> >>> restarted and shared lib is uploaded to hdfs as part > of > > >>> >> >> >> >> >>> deployment, hadoop will fail the > existing > > >>> jobs > > >>> >> >>for > > >>> >> >> >> >>which > > >>> >> >> >> >> >>> the > > >>> >> >> >> >> >>> timestamp of the file on hdfs doesn't match the > > >>> timestamp of > > >>> >> >>its > > >>> >> >> >> >>copy > > >>> >> >> >> >> >>> in > > >>> >> >> >> >> >>> the job's DC. > > >>> >> >> >> >> >>> > > >>> >> >> >> >> >>> > > >>> >> >> >> >> >>> Thanks, > > >>> >> >> >> >> >>> Virag > > >>> >> >> >> >> > > > >>> >> >> >> >> > > >>> >> >> >> > > >>> >> >> >> > > >>> >> >> > > >>> >> >> > > >>> >> > > >>> >> > > >>> > > > >>> > > >> > > >> > > > > > > > > > -- > > > Alejandro > > > > > >
