Tucu, That sounds good. Can we have the changes go in trunk first and once we validate that everything is fine, can we then port all the patches to branch-4.0?
Thanks, Rohini On Mon, May 6, 2013 at 2:52 PM, Alejandro Abdelnur <t...@cloudera.com>wrote: > Rohini, > > I understand you don't want to destabilize Oozie 4. Still, as we have no > done a 4 release yet, I think it is a good time to introduce this feature. > > How about the following (similar to what we did for command interruptions)? > > We do as Robert proposed in his last email, but keeping the default value > on OFF for Oozie 4; and in Oozie 5 we invert the default value to ON. > > This would mean you don't have to change anything in your deployment > protocol for Oozie 4. > > Thx > > > > On Mon, May 6, 2013 at 2:30 PM, Rohini Palaniswamy < > rohini.adi...@gmail.com> wrote: > >> Robert, >> > A new major release, like Oozie 4, would be a good time to introduce >> this change as the default; >> >> We already have Oozie 4 deployed in some clusters and it is stable >> now. We are waiting for users to validate Oozie-HCat integration before we >> deploy them in production clusters and call out a release in Apache. So it >> would be good to introduce this in the next release which will also be a >> big release anyway. >> >> Regards, >> Rohini >> >> >> On Mon, May 6, 2013 at 2:09 PM, Robert Kanter <rkan...@cloudera.com>wrote: >> >>> Sorry, I realize that I wasn't clear on that point. The Main classes >>> would >>> go in their respective sharelib (e.g. PigMain goes in share/lib/pig/) but >>> the other few classes that all actions use in the launcher jar (e.g. >>> LauncherMapper) would go in the oozie sharelib (share/lib/oozie), which >>> IIRC is already included when you use sharelibs (for example, when you >>> use >>> the sharelib with the Pig action, it includes both the share/lib/pig and >>> share/lib/oozie). >>> >>> thanks >>> - Robert >>> >>> >>> On Mon, May 6, 2013 at 1:52 PM, Virag Kothari <vi...@yahoo-inc.com> >>> wrote: >>> >>> > Hi Robert, >>> > >>> > I didn't understand completely. Are you suggesting to have all action >>> > specific main classes in /share/lib/oozie instead of their respective >>> > action sharelibs? >>> > >>> > Thanks, >>> > Virag >>> > >>> > From: Robert Kanter <rkan...@cloudera.com> >>> > Date: Monday, May 6, 2013 12:47 PM >>> > To: "dev@oozie.apache.org" <dev@oozie.apache.org>, Virag Kothari < >>> > vi...@yahoo-inc.com>, Alejandro Abdelnur <t...@cloudera.com> >>> > Subject: Re: OOZIE-1311 >>> > >>> > Virag, >>> > I talked more with Alejandro about the sharelib and the launcher jar. >>> > >>> > It would be good if we could eventually get rid of the launcher jar >>> > completely because: >>> > - less files would be copied to HDFS so it should be faster >>> > - it would fix an issue where some linux distress cleanup the temp dir >>> and >>> > delete the launcher jar >>> > >>> > The contents of the launcher jar could be moved to the Oozie sharelib >>> > (/share/lib/oozie) which IIRC currently only has a json jar file. >>> > >>> > As for backwards compatibility, we could do something like what you >>> > suggested for the other sharelibs where the webapp module includes >>> them. >>> > Then we'd add some property to oozie-site that would switch between >>> the >>> > old behavior with the launcher jar (sharelib is not required) and the >>> new >>> > behavior without the launcher jar (sharelib is required). A new major >>> > release, like Oozie 4, would be a good time to introduce this change >>> as the >>> > default; users can use the property to go back to the old behavior. >>> Also, >>> > if set to use the new behavior, we should make >>> oozie.use.system.libpath in >>> > the job.properties default to true because the sharelib would be >>> required, >>> > and if set to the old behavior, it would default to false. >>> > >>> > Thoughts? >>> > >>> > thanks >>> > - Robert >>> > >>> > >>> > >>> > >>> > >>> > On Wed, Apr 24, 2013 at 12:24 PM, Virag Kothari <vi...@yahoo-inc.com >>> >wrote: >>> > >>> >> Thanks Robert. Created OOZIE-1341 >>> >> >>> >> On 4/24/13 11:36 AM, "Robert Kanter" <rkan...@cloudera.com> wrote: >>> >> >>> >> >I think I follow what you're saying; assuming everything works >>> correctly, >>> >> >I >>> >> >think that sounds good. Can you make a JIRA with that and put up a >>> >> patch; >>> >> >that might be easier to see how this would work. >>> >> > >>> >> >thanks >>> >> >- Robert >>> >> > >>> >> > >>> >> >On Wed, Apr 24, 2013 at 11:29 AM, Virag Kothari <vi...@yahoo-inc.com >>> > >>> >> >wrote: >>> >> > >>> >> >> We can keep the Main classes in the share lib, but have the action >>> >> >> executors package the main classes for shipping it to >>> >> >> Launcher (basically again having the method getLauncherClasses() in >>> >> >>action >>> >> >> executors). Also the oozie-sharelib.jar containing main classes >>> >> >> should be part of web-app. >>> >> >> This will improve the building/testing env as classes are in >>> isolated >>> >> >> sharelibs but will make the share lib optional as main classes are >>> part >>> >> >>of >>> >> >> oozie-server. >>> >> >> Can we do this change to trunk as this is blocking our QE's to pick >>> >> >>builds >>> >> >> due to share lib incompatibility? >>> >> >> >>> >> >> We can still do what oozie-1318 is trying to achieve. For, e.g with >>> >> Hcat >>> >> >> integration, >>> >> >> there can be multiple overridable implementation of URIHandlers >>> that >>> >> can >>> >> >> be shipped to launcher. >>> >> >> We can do the same for other main classes if required. >>> >> >> >>> >> >> >>> >> >> Thanks, >>> >> >> Virag >>> >> >> >>> >> >> >>> >> >> On 4/24/13 10:18 AM, "Robert Kanter" <rkan...@cloudera.com> wrote: >>> >> >> >>> >> >> >That's true, its primarily a building/testing improvement. And >>> from a >>> >> >> >design perspective, it could be considered "cleaner" to not have >>> the >>> >> >> >launcher jar. >>> >> >> > >>> >> >> >Your idea sounds like a good compromise between backwards >>> >> compatibility >>> >> >> >and >>> >> >> >improving the building/testing env. How would we do that? >>> >> >> > >>> >> >> >As a side note, there's also a new feature with the Main classes >>> being >>> >> >>in >>> >> >> >the sharelib; users can now override the Main class that Oozie >>> will >>> >> use >>> >> >> >for >>> >> >> >the action because its no longer in the launcher jar. I think >>> this >>> >> >>could >>> >> >> >still be a useful feature, but I'm guessing that if we do the >>> >> >>compromise, >>> >> >> >this would go away? In any case, OOZIE-1318 is to make the Main >>> >> >> >overridable with a config, which is probably cleaner and more >>> flexible >>> >> >> >than >>> >> >> >replacing a jar file. >>> >> >> > >>> >> >> >thanks >>> >> >> >- Robert >>> >> >> > >>> >> >> > >>> >> >> >On Wed, Apr 24, 2013 at 9:54 AM, Virag Kothari < >>> vi...@yahoo-inc.com> >>> >> >> >wrote: >>> >> >> > >>> >> >> >> That is a good suggestion for preserving compatibility, Robert. >>> >> >> >> >>> >> >> >> Another question is what is the main purpose of OOZIE-1311? From >>> >> JIRA >>> >> >> >> description, the primary >>> >> >> >> advantage of moving Main classes from core to share lib is it >>> can >>> >> >>help >>> >> >> >>in >>> >> >> >> preventing dependencies conflicts >>> >> >> >> (different versions of antlr's for pig, hive). However, that >>> problem >>> >> >> >>only >>> >> >> >> exists while building/testing Oozie. >>> >> >> >> If that is the case, main classes can be moved to share lib but >>> they >>> >> >>can >>> >> >> >> be bundled in oozie-server itself. >>> >> >> >> In that case, the deployment doesn't change and share lib can >>> still >>> >> >>be >>> >> >> >> optional >>> >> >> >> as oozie server would ship those classes (same as what was >>> happening >>> >> >> >> before). And, >>> >> >> >> advantage of 1311 for building/testing will be retained >>> >> >> >> >>> >> >> >> The main concern is having share lib as a required installation >>> will >>> >> >> >> complicate deployment. >>> >> >> >> Doing hot upgrade for share lib is non-trivial and Oozie doesn't >>> >> >>provide >>> >> >> >> inbuilt support for that. >>> >> >> >> Even though end-users wouldn't notice the change, it will be a >>> big >>> >> >> >>change >>> >> >> >> for system admins installing oozie >>> >> >> >> So even though share lib is nice feature to have, keeping it >>> >> optional >>> >> >> >> seems better to me as there are other ways >>> >> >> >> Of including jars and which were introduced much before share >>> lib. >>> >> >> >> >>> >> >> >> Thanks, >>> >> >> >> Virag >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> On 4/24/13 9:09 AM, "Robert Kanter" <rkan...@cloudera.com> >>> wrote: >>> >> >> >> >>> >> >> >> >Alejandro had another suggestion to prevent breaking stuff for >>> >> >>existing >>> >> >> >> >apps where the sharelib isn't used. Assuming the sharelib is >>> >> >> >>installed, >>> >> >> >> >if >>> >> >> >> >the user doesn't set oozie.use.system.libpath to true, then >>> Oozie >>> >> >>would >>> >> >> >> >only include the oozie-*.jar from the sharelib (the one with >>> the >>> >> >>Main) >>> >> >> >>in >>> >> >> >> >it; if its set to true, then Oozie would include all of the >>> jars in >>> >> >>the >>> >> >> >> >sharelib. This would allow users to still use their own lib >>> dir >>> >> >>while >>> >> >> >> >still getting the Main into the classpath. >>> >> >> >> > >>> >> >> >> >- Robert >>> >> >> >> > >>> >> >> >> > >>> >> >> >> > >>> >> >> >> >On Tue, Apr 23, 2013 at 10:46 PM, Alejandro Abdelnur >>> >> >> >> ><tuc...@gmail.com>wrote: >>> >> >> >> > >>> >> >> >> >> virag, if your oozie servers dont have sharelibs currently >>> >> >>installed >>> >> >> >> >>then >>> >> >> >> >> unstalling sharelibs as part of an update would not break ant >>> >> >>running >>> >> >> >> >>app. >>> >> >> >> >> regarding the requirement of enabling the sharelibs to have >>> the >>> >> >> >>mains, >>> >> >> >> >>you >>> >> >> >> >> have a point. we need to figure out how to get those in the >>> >> >> >>classpath if >>> >> >> >> >> the user is not using the sharelibs. i'd suggest we leave >>> this >>> >> >> >> >> patch in trunk for now and we move it to a release branch >>> once we >>> >> >> >>sorted >>> >> >> >> >> out a way of solving this incompatibility. >>> >> >> >> >> >>> >> >> >> >> thanks >>> >> >> >> >> >>> >> >> >> >> Alejandro >>> >> >> >> >> (phone typing) >>> >> >> >> >> >>> >> >> >> >> On Apr 23, 2013, at 5:09 PM, Virag Kothari < >>> vi...@yahoo-inc.com> >>> >> >> >>wrote: >>> >> >> >> >> >>> >> >> >> >> > >>> >> >> >> >> > Tucu, Thanks for the reply and explanation. >>> >> >> >> >> > For workflows running in Y!, cold upgrade is not a feasible >>> >> >>option >>> >> >> >>as >>> >> >> >> >>we >>> >> >> >> >> > cannot expect all the running actions to complete before >>> Oozie >>> >> >> >> >>restart. >>> >> >> >> >> > Hot upgrade is a good option, but we will need time to >>> >> >>implement it >>> >> >> >> >>as we >>> >> >> >> >> > currently don't have sharelib as part of our deployment. >>> >> >> >> >> > So can we deprecate the refactoring of launcher classes in >>> this >>> >> >> >> >>release? >>> >> >> >> >> > Also, all the launcher classes are only added to >>> distributed >>> >> >>cache >>> >> >> >>if >>> >> >> >> >>the >>> >> >> >> >> > workflow is configured to use system lib path >>> >> >> >> >> > (oozie.use.system.libpath=true) >>> >> >> >> >> > We shouldn't mandate the workflow to have this property as >>> they >>> >> >>are >>> >> >> >> >> > oozie's launcher classes. >>> >> >> >> >> > >>> >> >> >> >> > Thanks, >>> >> >> >> >> > Virag >>> >> >> >> >> > >>> >> >> >> >> > >>> >> >> >> >> > >>> >> >> >> >> > On 4/23/13 3:21 PM, "Alejandro Abdelnur" <tuc...@gmail.com >>> > >>> >> >>wrote: >>> >> >> >> >> > >>> >> >> >> >> >> Virag, >>> >> >> >> >> >> >>> >> >> >> >> >> On sharelib being required, yes you are correct. For this >>> we >>> >> >> >>should: >>> >> >> >> >> >> >>> >> >> >> >> >> * Make 100% clear in the quick-start/install docs that the >>> >> >> >>sharelib >>> >> >> >> >>is >>> >> >> >> >> >> REQUIRED. >>> >> >> >> >> >> >>> >> >> >> >> >> * Add a check at Oozie startup to verify the sharelib dir >>> >> >>exists, >>> >> >> >> >>else >>> >> >> >> >> >> fail >>> >> >> >> >> >> to start. >>> >> >> >> >> >> >>> >> >> >> >> >> On sharelib lib issue during upgrade for in-flight jobs. >>> >> >> >>Depending on >>> >> >> >> >> the >>> >> >> >> >> >> type of upgrade this may be an issue. >>> >> >> >> >> >> >>> >> >> >> >> >> * If you are upgrading an oozie server fix that does not >>> >> change >>> >> >> >>the >>> >> >> >> >> >> sharelib files, this is not an issue and you can just >>> shutdown >>> >> >>the >>> >> >> >> >>oozie >>> >> >> >> >> >> server with in-flight jobs. >>> >> >> >> >> >> >>> >> >> >> >> >> * If you are upgrading an oozie server fix that involves >>> >> >>sharelib >>> >> >> >> >>files >>> >> >> >> >> >> changes, then you have 2 options: >>> >> >> >> >> >> >>> >> >> >> >> >> ** Cold upgrade: bring all WF jobs to suspend/completion, >>> wait >>> >> >> >>till >>> >> >> >> >>all >>> >> >> >> >> >> running actions end, then shutdown oozie server, upgrade >>> oozie >>> >> >> >>server >>> >> >> >> >> and >>> >> >> >> >> >> sharelib. Then restart oozie server and resume WF jobs. In >>> >> this >>> >> >> >>case >>> >> >> >> >>all >>> >> >> >> >> >> new WF actions will use the new sharelib. >>> >> >> >> >> >> >>> >> >> >> >> >> ** Hot upgrade: stop oozie server. modify the >>> oozie-site.xml >>> >> >> >>sharelib >>> >> >> >> >> >> location to point to a new directory. upgrade the oozie >>> >> server. >>> >> >> >> >>install >>> >> >> >> >> >> the >>> >> >> >> >> >> sharelib (will be a create as the sharelib dir in HDFS >>> does >>> >> not >>> >> >> >> >>exist). >>> >> >> >> >> >> start the oozie server. In this case all running WF >>> actions >>> >> >>will >>> >> >> >> >> continue >>> >> >> >> >> >> running with no issues as the JARs in the distributed >>> cache >>> >> >>have >>> >> >> >>not >>> >> >> >> >> been >>> >> >> >> >> >> touch. All new WF actions will start using the new >>> sharelib. >>> >> >> >> >> >> >>> >> >> >> >> >> Note that this sharelib upgrade protocol is not >>> introduced by >>> >> >> >> >>requiring >>> >> >> >> >> >> sharelib, it is required if you have applications that use >>> >> >> >>sharelib >>> >> >> >> >> today. >>> >> >> >> >> >> >>> >> >> >> >> >> Does this address your concerns? >>> >> >> >> >> >> >>> >> >> >> >> >> Thanks. >>> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >> On Tue, Apr 23, 2013 at 1:35 PM, Virag Kothari >>> >> >> >><vi...@yahoo-inc.com> >>> >> >> >> >> >> wrote: >>> >> >> >> >> >> >>> >> >> >> >> >>> Hi, >>> >> >> >> >> >>> >>> >> >> >> >> >>> With OOZIE-1311 and its subtasks, the idea seems to >>> move all >>> >> >>the >>> >> >> >> >> >>> launcher >>> >> >> >> >> >>> classes like PigMain, HiveMain etc. to their respective >>> >> >> >>sharelibs. >>> >> >> >> >> >>> So, now shared lib is a mandatory deployment step. Before >>> >> >>shared >>> >> >> >>lib >>> >> >> >> >> was >>> >> >> >> >> >>> optional as users could bundle jars with their workflow >>> >> >> >>application. >>> >> >> >> >> >>> So always requiring shared lib seems to introduce 2 >>> problems: >>> >> >> >> >> >>> >>> >> >> >> >> >>> 1. The current deployments which don't use action >>> shared >>> >> lib >>> >> >> >>will >>> >> >> >> >> >>> fail. >>> >> >> >> >> >>> So, probably we should deprecate the current behavior. >>> >> >> >> >> >>> >>> >> >> >> >> >>> 2. The hadoop distributed cache mechanism will fail a >>> job if >>> >> >>the >>> >> >> >> >>files >>> >> >> >> >> >>> in >>> >> >> >> >> >>> DC are updated on hdfs while the hadoop job is running. >>> So, >>> >> >>when >>> >> >> >> >>Oozie >>> >> >> >> >> >>> is >>> >> >> >> >> >>> restarted and shared lib is uploaded to hdfs as part of >>> >> >> >> >> >>> deployment, hadoop will fail the existing >>> jobs >>> >> >>for >>> >> >> >> >>which >>> >> >> >> >> >>> the >>> >> >> >> >> >>> timestamp of the file on hdfs doesn't match the >>> timestamp of >>> >> >>its >>> >> >> >> >>copy >>> >> >> >> >> >>> in >>> >> >> >> >> >>> the job's DC. >>> >> >> >> >> >>> >>> >> >> >> >> >>> >>> >> >> >> >> >>> Thanks, >>> >> >> >> >> >>> Virag >>> >> >> >> >> > >>> >> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >>> >> >> >>> >> >>> >> >>> > >>> >> >> > > > -- > Alejandro >
