We can keep the Main classes in the share lib, but have the action executors package the main classes for shipping it to Launcher (basically again having the method getLauncherClasses() in action executors). Also the oozie-sharelib.jar containing main classes should be part of web-app. This will improve the building/testing env as classes are in isolated sharelibs but will make the share lib optional as main classes are part of oozie-server. Can we do this change to trunk as this is blocking our QE's to pick builds due to share lib incompatibility?
We can still do what oozie-1318 is trying to achieve. For, e.g with Hcat integration, there can be multiple overridable implementation of URIHandlers that can be shipped to launcher. We can do the same for other main classes if required. Thanks, Virag On 4/24/13 10:18 AM, "Robert Kanter" <rkan...@cloudera.com> wrote: >That's true, its primarily a building/testing improvement. And from a >design perspective, it could be considered "cleaner" to not have the >launcher jar. > >Your idea sounds like a good compromise between backwards compatibility >and >improving the building/testing env. How would we do that? > >As a side note, there's also a new feature with the Main classes being in >the sharelib; users can now override the Main class that Oozie will use >for >the action because its no longer in the launcher jar. I think this could >still be a useful feature, but I'm guessing that if we do the compromise, >this would go away? In any case, OOZIE-1318 is to make the Main >overridable with a config, which is probably cleaner and more flexible >than >replacing a jar file. > >thanks >- Robert > > >On Wed, Apr 24, 2013 at 9:54 AM, Virag Kothari <vi...@yahoo-inc.com> >wrote: > >> That is a good suggestion for preserving compatibility, Robert. >> >> Another question is what is the main purpose of OOZIE-1311? From JIRA >> description, the primary >> advantage of moving Main classes from core to share lib is it can help >>in >> preventing dependencies conflicts >> (different versions of antlr's for pig, hive). However, that problem >>only >> exists while building/testing Oozie. >> If that is the case, main classes can be moved to share lib but they can >> be bundled in oozie-server itself. >> In that case, the deployment doesn't change and share lib can still be >> optional >> as oozie server would ship those classes (same as what was happening >> before). And, >> advantage of 1311 for building/testing will be retained >> >> The main concern is having share lib as a required installation will >> complicate deployment. >> Doing hot upgrade for share lib is non-trivial and Oozie doesn't provide >> inbuilt support for that. >> Even though end-users wouldn't notice the change, it will be a big >>change >> for system admins installing oozie >> So even though share lib is nice feature to have, keeping it optional >> seems better to me as there are other ways >> Of including jars and which were introduced much before share lib. >> >> Thanks, >> Virag >> >> >> >> >> On 4/24/13 9:09 AM, "Robert Kanter" <rkan...@cloudera.com> wrote: >> >> >Alejandro had another suggestion to prevent breaking stuff for existing >> >apps where the sharelib isn't used. Assuming the sharelib is >>installed, >> >if >> >the user doesn't set oozie.use.system.libpath to true, then Oozie would >> >only include the oozie-*.jar from the sharelib (the one with the Main) >>in >> >it; if its set to true, then Oozie would include all of the jars in the >> >sharelib. This would allow users to still use their own lib dir while >> >still getting the Main into the classpath. >> > >> >- Robert >> > >> > >> > >> >On Tue, Apr 23, 2013 at 10:46 PM, Alejandro Abdelnur >> ><tuc...@gmail.com>wrote: >> > >> >> virag, if your oozie servers dont have sharelibs currently installed >> >>then >> >> unstalling sharelibs as part of an update would not break ant running >> >>app. >> >> regarding the requirement of enabling the sharelibs to have the >>mains, >> >>you >> >> have a point. we need to figure out how to get those in the >>classpath if >> >> the user is not using the sharelibs. i'd suggest we leave this >> >> patch in trunk for now and we move it to a release branch once we >>sorted >> >> out a way of solving this incompatibility. >> >> >> >> thanks >> >> >> >> Alejandro >> >> (phone typing) >> >> >> >> On Apr 23, 2013, at 5:09 PM, Virag Kothari <vi...@yahoo-inc.com> >>wrote: >> >> >> >> > >> >> > Tucu, Thanks for the reply and explanation. >> >> > For workflows running in Y!, cold upgrade is not a feasible option >>as >> >>we >> >> > cannot expect all the running actions to complete before Oozie >> >>restart. >> >> > Hot upgrade is a good option, but we will need time to implement it >> >>as we >> >> > currently don't have sharelib as part of our deployment. >> >> > So can we deprecate the refactoring of launcher classes in this >> >>release? >> >> > Also, all the launcher classes are only added to distributed cache >>if >> >>the >> >> > workflow is configured to use system lib path >> >> > (oozie.use.system.libpath=true) >> >> > We shouldn't mandate the workflow to have this property as they are >> >> > oozie's launcher classes. >> >> > >> >> > Thanks, >> >> > Virag >> >> > >> >> > >> >> > >> >> > On 4/23/13 3:21 PM, "Alejandro Abdelnur" <tuc...@gmail.com> wrote: >> >> > >> >> >> Virag, >> >> >> >> >> >> On sharelib being required, yes you are correct. For this we >>should: >> >> >> >> >> >> * Make 100% clear in the quick-start/install docs that the >>sharelib >> >>is >> >> >> REQUIRED. >> >> >> >> >> >> * Add a check at Oozie startup to verify the sharelib dir exists, >> >>else >> >> >> fail >> >> >> to start. >> >> >> >> >> >> On sharelib lib issue during upgrade for in-flight jobs. >>Depending on >> >> the >> >> >> type of upgrade this may be an issue. >> >> >> >> >> >> * If you are upgrading an oozie server fix that does not change >>the >> >> >> sharelib files, this is not an issue and you can just shutdown the >> >>oozie >> >> >> server with in-flight jobs. >> >> >> >> >> >> * If you are upgrading an oozie server fix that involves sharelib >> >>files >> >> >> changes, then you have 2 options: >> >> >> >> >> >> ** Cold upgrade: bring all WF jobs to suspend/completion, wait >>till >> >>all >> >> >> running actions end, then shutdown oozie server, upgrade oozie >>server >> >> and >> >> >> sharelib. Then restart oozie server and resume WF jobs. In this >>case >> >>all >> >> >> new WF actions will use the new sharelib. >> >> >> >> >> >> ** Hot upgrade: stop oozie server. modify the oozie-site.xml >>sharelib >> >> >> location to point to a new directory. upgrade the oozie server. >> >>install >> >> >> the >> >> >> sharelib (will be a create as the sharelib dir in HDFS does not >> >>exist). >> >> >> start the oozie server. In this case all running WF actions will >> >> continue >> >> >> running with no issues as the JARs in the distributed cache have >>not >> >> been >> >> >> touch. All new WF actions will start using the new sharelib. >> >> >> >> >> >> Note that this sharelib upgrade protocol is not introduced by >> >>requiring >> >> >> sharelib, it is required if you have applications that use >>sharelib >> >> today. >> >> >> >> >> >> Does this address your concerns? >> >> >> >> >> >> Thanks. >> >> >> >> >> >> >> >> >> >> >> >> On Tue, Apr 23, 2013 at 1:35 PM, Virag Kothari >><vi...@yahoo-inc.com> >> >> >> wrote: >> >> >> >> >> >>> Hi, >> >> >>> >> >> >>> With OOZIE-1311 and its subtasks, the idea seems to move all the >> >> >>> launcher >> >> >>> classes like PigMain, HiveMain etc. to their respective >>sharelibs. >> >> >>> So, now shared lib is a mandatory deployment step. Before shared >>lib >> >> was >> >> >>> optional as users could bundle jars with their workflow >>application. >> >> >>> So always requiring shared lib seems to introduce 2 problems: >> >> >>> >> >> >>> 1. The current deployments which don't use action shared lib >>will >> >> >>> fail. >> >> >>> So, probably we should deprecate the current behavior. >> >> >>> >> >> >>> 2. The hadoop distributed cache mechanism will fail a job if the >> >>files >> >> >>> in >> >> >>> DC are updated on hdfs while the hadoop job is running. So, when >> >>Oozie >> >> >>> is >> >> >>> restarted and shared lib is uploaded to hdfs as part of >> >> >>> deployment, hadoop will fail the existing jobs for >> >>which >> >> >>> the >> >> >>> timestamp of the file on hdfs doesn't match the timestamp of its >> >>copy >> >> >>> in >> >> >>> the job's DC. >> >> >>> >> >> >>> >> >> >>> Thanks, >> >> >>> Virag >> >> > >> >> >> >>
