Robert, > A new major release, like Oozie 4, would be a good time to introduce this change as the default;
We already have Oozie 4 deployed in some clusters and it is stable now. We are waiting for users to validate Oozie-HCat integration before we deploy them in production clusters and call out a release in Apache. So it would be good to introduce this in the next release which will also be a big release anyway. Regards, Rohini On Mon, May 6, 2013 at 2:09 PM, Robert Kanter <rkan...@cloudera.com> wrote: > Sorry, I realize that I wasn't clear on that point. The Main classes would > go in their respective sharelib (e.g. PigMain goes in share/lib/pig/) but > the other few classes that all actions use in the launcher jar (e.g. > LauncherMapper) would go in the oozie sharelib (share/lib/oozie), which > IIRC is already included when you use sharelibs (for example, when you use > the sharelib with the Pig action, it includes both the share/lib/pig and > share/lib/oozie). > > thanks > - Robert > > > On Mon, May 6, 2013 at 1:52 PM, Virag Kothari <vi...@yahoo-inc.com> wrote: > > > Hi Robert, > > > > I didn't understand completely. Are you suggesting to have all action > > specific main classes in /share/lib/oozie instead of their respective > > action sharelibs? > > > > Thanks, > > Virag > > > > From: Robert Kanter <rkan...@cloudera.com> > > Date: Monday, May 6, 2013 12:47 PM > > To: "dev@oozie.apache.org" <dev@oozie.apache.org>, Virag Kothari < > > vi...@yahoo-inc.com>, Alejandro Abdelnur <t...@cloudera.com> > > Subject: Re: OOZIE-1311 > > > > Virag, > > I talked more with Alejandro about the sharelib and the launcher jar. > > > > It would be good if we could eventually get rid of the launcher jar > > completely because: > > - less files would be copied to HDFS so it should be faster > > - it would fix an issue where some linux distress cleanup the temp dir > and > > delete the launcher jar > > > > The contents of the launcher jar could be moved to the Oozie sharelib > > (/share/lib/oozie) which IIRC currently only has a json jar file. > > > > As for backwards compatibility, we could do something like what you > > suggested for the other sharelibs where the webapp module includes them. > > Then we'd add some property to oozie-site that would switch between the > > old behavior with the launcher jar (sharelib is not required) and the new > > behavior without the launcher jar (sharelib is required). A new major > > release, like Oozie 4, would be a good time to introduce this change as > the > > default; users can use the property to go back to the old behavior. > Also, > > if set to use the new behavior, we should make oozie.use.system.libpath > in > > the job.properties default to true because the sharelib would be > required, > > and if set to the old behavior, it would default to false. > > > > Thoughts? > > > > thanks > > - Robert > > > > > > > > > > > > On Wed, Apr 24, 2013 at 12:24 PM, Virag Kothari <vi...@yahoo-inc.com > >wrote: > > > >> Thanks Robert. Created OOZIE-1341 > >> > >> On 4/24/13 11:36 AM, "Robert Kanter" <rkan...@cloudera.com> wrote: > >> > >> >I think I follow what you're saying; assuming everything works > correctly, > >> >I > >> >think that sounds good. Can you make a JIRA with that and put up a > >> patch; > >> >that might be easier to see how this would work. > >> > > >> >thanks > >> >- Robert > >> > > >> > > >> >On Wed, Apr 24, 2013 at 11:29 AM, Virag Kothari <vi...@yahoo-inc.com> > >> >wrote: > >> > > >> >> We can keep the Main classes in the share lib, but have the action > >> >> executors package the main classes for shipping it to > >> >> Launcher (basically again having the method getLauncherClasses() in > >> >>action > >> >> executors). Also the oozie-sharelib.jar containing main classes > >> >> should be part of web-app. > >> >> This will improve the building/testing env as classes are in isolated > >> >> sharelibs but will make the share lib optional as main classes are > part > >> >>of > >> >> oozie-server. > >> >> Can we do this change to trunk as this is blocking our QE's to pick > >> >>builds > >> >> due to share lib incompatibility? > >> >> > >> >> We can still do what oozie-1318 is trying to achieve. For, e.g with > >> Hcat > >> >> integration, > >> >> there can be multiple overridable implementation of URIHandlers that > >> can > >> >> be shipped to launcher. > >> >> We can do the same for other main classes if required. > >> >> > >> >> > >> >> Thanks, > >> >> Virag > >> >> > >> >> > >> >> On 4/24/13 10:18 AM, "Robert Kanter" <rkan...@cloudera.com> wrote: > >> >> > >> >> >That's true, its primarily a building/testing improvement. And > from a > >> >> >design perspective, it could be considered "cleaner" to not have the > >> >> >launcher jar. > >> >> > > >> >> >Your idea sounds like a good compromise between backwards > >> compatibility > >> >> >and > >> >> >improving the building/testing env. How would we do that? > >> >> > > >> >> >As a side note, there's also a new feature with the Main classes > being > >> >>in > >> >> >the sharelib; users can now override the Main class that Oozie will > >> use > >> >> >for > >> >> >the action because its no longer in the launcher jar. I think this > >> >>could > >> >> >still be a useful feature, but I'm guessing that if we do the > >> >>compromise, > >> >> >this would go away? In any case, OOZIE-1318 is to make the Main > >> >> >overridable with a config, which is probably cleaner and more > flexible > >> >> >than > >> >> >replacing a jar file. > >> >> > > >> >> >thanks > >> >> >- Robert > >> >> > > >> >> > > >> >> >On Wed, Apr 24, 2013 at 9:54 AM, Virag Kothari <vi...@yahoo-inc.com > > > >> >> >wrote: > >> >> > > >> >> >> That is a good suggestion for preserving compatibility, Robert. > >> >> >> > >> >> >> Another question is what is the main purpose of OOZIE-1311? From > >> JIRA > >> >> >> description, the primary > >> >> >> advantage of moving Main classes from core to share lib is it can > >> >>help > >> >> >>in > >> >> >> preventing dependencies conflicts > >> >> >> (different versions of antlr's for pig, hive). However, that > problem > >> >> >>only > >> >> >> exists while building/testing Oozie. > >> >> >> If that is the case, main classes can be moved to share lib but > they > >> >>can > >> >> >> be bundled in oozie-server itself. > >> >> >> In that case, the deployment doesn't change and share lib can > still > >> >>be > >> >> >> optional > >> >> >> as oozie server would ship those classes (same as what was > happening > >> >> >> before). And, > >> >> >> advantage of 1311 for building/testing will be retained > >> >> >> > >> >> >> The main concern is having share lib as a required installation > will > >> >> >> complicate deployment. > >> >> >> Doing hot upgrade for share lib is non-trivial and Oozie doesn't > >> >>provide > >> >> >> inbuilt support for that. > >> >> >> Even though end-users wouldn't notice the change, it will be a big > >> >> >>change > >> >> >> for system admins installing oozie > >> >> >> So even though share lib is nice feature to have, keeping it > >> optional > >> >> >> seems better to me as there are other ways > >> >> >> Of including jars and which were introduced much before share lib. > >> >> >> > >> >> >> Thanks, > >> >> >> Virag > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> On 4/24/13 9:09 AM, "Robert Kanter" <rkan...@cloudera.com> wrote: > >> >> >> > >> >> >> >Alejandro had another suggestion to prevent breaking stuff for > >> >>existing > >> >> >> >apps where the sharelib isn't used. Assuming the sharelib is > >> >> >>installed, > >> >> >> >if > >> >> >> >the user doesn't set oozie.use.system.libpath to true, then Oozie > >> >>would > >> >> >> >only include the oozie-*.jar from the sharelib (the one with the > >> >>Main) > >> >> >>in > >> >> >> >it; if its set to true, then Oozie would include all of the jars > in > >> >>the > >> >> >> >sharelib. This would allow users to still use their own lib dir > >> >>while > >> >> >> >still getting the Main into the classpath. > >> >> >> > > >> >> >> >- Robert > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> >On Tue, Apr 23, 2013 at 10:46 PM, Alejandro Abdelnur > >> >> >> ><tuc...@gmail.com>wrote: > >> >> >> > > >> >> >> >> virag, if your oozie servers dont have sharelibs currently > >> >>installed > >> >> >> >>then > >> >> >> >> unstalling sharelibs as part of an update would not break ant > >> >>running > >> >> >> >>app. > >> >> >> >> regarding the requirement of enabling the sharelibs to have the > >> >> >>mains, > >> >> >> >>you > >> >> >> >> have a point. we need to figure out how to get those in the > >> >> >>classpath if > >> >> >> >> the user is not using the sharelibs. i'd suggest we leave this > >> >> >> >> patch in trunk for now and we move it to a release branch once > we > >> >> >>sorted > >> >> >> >> out a way of solving this incompatibility. > >> >> >> >> > >> >> >> >> thanks > >> >> >> >> > >> >> >> >> Alejandro > >> >> >> >> (phone typing) > >> >> >> >> > >> >> >> >> On Apr 23, 2013, at 5:09 PM, Virag Kothari < > vi...@yahoo-inc.com> > >> >> >>wrote: > >> >> >> >> > >> >> >> >> > > >> >> >> >> > Tucu, Thanks for the reply and explanation. > >> >> >> >> > For workflows running in Y!, cold upgrade is not a feasible > >> >>option > >> >> >>as > >> >> >> >>we > >> >> >> >> > cannot expect all the running actions to complete before > Oozie > >> >> >> >>restart. > >> >> >> >> > Hot upgrade is a good option, but we will need time to > >> >>implement it > >> >> >> >>as we > >> >> >> >> > currently don't have sharelib as part of our deployment. > >> >> >> >> > So can we deprecate the refactoring of launcher classes in > this > >> >> >> >>release? > >> >> >> >> > Also, all the launcher classes are only added to distributed > >> >>cache > >> >> >>if > >> >> >> >>the > >> >> >> >> > workflow is configured to use system lib path > >> >> >> >> > (oozie.use.system.libpath=true) > >> >> >> >> > We shouldn't mandate the workflow to have this property as > they > >> >>are > >> >> >> >> > oozie's launcher classes. > >> >> >> >> > > >> >> >> >> > Thanks, > >> >> >> >> > Virag > >> >> >> >> > > >> >> >> >> > > >> >> >> >> > > >> >> >> >> > On 4/23/13 3:21 PM, "Alejandro Abdelnur" <tuc...@gmail.com> > >> >>wrote: > >> >> >> >> > > >> >> >> >> >> Virag, > >> >> >> >> >> > >> >> >> >> >> On sharelib being required, yes you are correct. For this we > >> >> >>should: > >> >> >> >> >> > >> >> >> >> >> * Make 100% clear in the quick-start/install docs that the > >> >> >>sharelib > >> >> >> >>is > >> >> >> >> >> REQUIRED. > >> >> >> >> >> > >> >> >> >> >> * Add a check at Oozie startup to verify the sharelib dir > >> >>exists, > >> >> >> >>else > >> >> >> >> >> fail > >> >> >> >> >> to start. > >> >> >> >> >> > >> >> >> >> >> On sharelib lib issue during upgrade for in-flight jobs. > >> >> >>Depending on > >> >> >> >> the > >> >> >> >> >> type of upgrade this may be an issue. > >> >> >> >> >> > >> >> >> >> >> * If you are upgrading an oozie server fix that does not > >> change > >> >> >>the > >> >> >> >> >> sharelib files, this is not an issue and you can just > shutdown > >> >>the > >> >> >> >>oozie > >> >> >> >> >> server with in-flight jobs. > >> >> >> >> >> > >> >> >> >> >> * If you are upgrading an oozie server fix that involves > >> >>sharelib > >> >> >> >>files > >> >> >> >> >> changes, then you have 2 options: > >> >> >> >> >> > >> >> >> >> >> ** Cold upgrade: bring all WF jobs to suspend/completion, > wait > >> >> >>till > >> >> >> >>all > >> >> >> >> >> running actions end, then shutdown oozie server, upgrade > oozie > >> >> >>server > >> >> >> >> and > >> >> >> >> >> sharelib. Then restart oozie server and resume WF jobs. In > >> this > >> >> >>case > >> >> >> >>all > >> >> >> >> >> new WF actions will use the new sharelib. > >> >> >> >> >> > >> >> >> >> >> ** Hot upgrade: stop oozie server. modify the oozie-site.xml > >> >> >>sharelib > >> >> >> >> >> location to point to a new directory. upgrade the oozie > >> server. > >> >> >> >>install > >> >> >> >> >> the > >> >> >> >> >> sharelib (will be a create as the sharelib dir in HDFS does > >> not > >> >> >> >>exist). > >> >> >> >> >> start the oozie server. In this case all running WF actions > >> >>will > >> >> >> >> continue > >> >> >> >> >> running with no issues as the JARs in the distributed cache > >> >>have > >> >> >>not > >> >> >> >> been > >> >> >> >> >> touch. All new WF actions will start using the new sharelib. > >> >> >> >> >> > >> >> >> >> >> Note that this sharelib upgrade protocol is not introduced > by > >> >> >> >>requiring > >> >> >> >> >> sharelib, it is required if you have applications that use > >> >> >>sharelib > >> >> >> >> today. > >> >> >> >> >> > >> >> >> >> >> Does this address your concerns? > >> >> >> >> >> > >> >> >> >> >> Thanks. > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> On Tue, Apr 23, 2013 at 1:35 PM, Virag Kothari > >> >> >><vi...@yahoo-inc.com> > >> >> >> >> >> wrote: > >> >> >> >> >> > >> >> >> >> >>> Hi, > >> >> >> >> >>> > >> >> >> >> >>> With OOZIE-1311 and its subtasks, the idea seems to move > all > >> >>the > >> >> >> >> >>> launcher > >> >> >> >> >>> classes like PigMain, HiveMain etc. to their respective > >> >> >>sharelibs. > >> >> >> >> >>> So, now shared lib is a mandatory deployment step. Before > >> >>shared > >> >> >>lib > >> >> >> >> was > >> >> >> >> >>> optional as users could bundle jars with their workflow > >> >> >>application. > >> >> >> >> >>> So always requiring shared lib seems to introduce 2 > problems: > >> >> >> >> >>> > >> >> >> >> >>> 1. The current deployments which don't use action shared > >> lib > >> >> >>will > >> >> >> >> >>> fail. > >> >> >> >> >>> So, probably we should deprecate the current behavior. > >> >> >> >> >>> > >> >> >> >> >>> 2. The hadoop distributed cache mechanism will fail a job > if > >> >>the > >> >> >> >>files > >> >> >> >> >>> in > >> >> >> >> >>> DC are updated on hdfs while the hadoop job is running. So, > >> >>when > >> >> >> >>Oozie > >> >> >> >> >>> is > >> >> >> >> >>> restarted and shared lib is uploaded to hdfs as part of > >> >> >> >> >>> deployment, hadoop will fail the existing jobs > >> >>for > >> >> >> >>which > >> >> >> >> >>> the > >> >> >> >> >>> timestamp of the file on hdfs doesn't match the timestamp > of > >> >>its > >> >> >> >>copy > >> >> >> >> >>> in > >> >> >> >> >>> the job's DC. > >> >> >> >> >>> > >> >> >> >> >>> > >> >> >> >> >>> Thanks, > >> >> >> >> >>> Virag > >> >> >> >> > > >> >> >> >> > >> >> >> > >> >> >> > >> >> > >> >> > >> > >> > > >
