That's true, its primarily a building/testing improvement. And from a design perspective, it could be considered "cleaner" to not have the launcher jar.
Your idea sounds like a good compromise between backwards compatibility and improving the building/testing env. How would we do that? As a side note, there's also a new feature with the Main classes being in the sharelib; users can now override the Main class that Oozie will use for the action because its no longer in the launcher jar. I think this could still be a useful feature, but I'm guessing that if we do the compromise, this would go away? In any case, OOZIE-1318 is to make the Main overridable with a config, which is probably cleaner and more flexible than replacing a jar file. thanks - Robert On Wed, Apr 24, 2013 at 9:54 AM, Virag Kothari <vi...@yahoo-inc.com> wrote: > That is a good suggestion for preserving compatibility, Robert. > > Another question is what is the main purpose of OOZIE-1311? From JIRA > description, the primary > advantage of moving Main classes from core to share lib is it can help in > preventing dependencies conflicts > (different versions of antlr's for pig, hive). However, that problem only > exists while building/testing Oozie. > If that is the case, main classes can be moved to share lib but they can > be bundled in oozie-server itself. > In that case, the deployment doesn't change and share lib can still be > optional > as oozie server would ship those classes (same as what was happening > before). And, > advantage of 1311 for building/testing will be retained > > The main concern is having share lib as a required installation will > complicate deployment. > Doing hot upgrade for share lib is non-trivial and Oozie doesn't provide > inbuilt support for that. > Even though end-users wouldn't notice the change, it will be a big change > for system admins installing oozie > So even though share lib is nice feature to have, keeping it optional > seems better to me as there are other ways > Of including jars and which were introduced much before share lib. > > Thanks, > Virag > > > > > On 4/24/13 9:09 AM, "Robert Kanter" <rkan...@cloudera.com> wrote: > > >Alejandro had another suggestion to prevent breaking stuff for existing > >apps where the sharelib isn't used. Assuming the sharelib is installed, > >if > >the user doesn't set oozie.use.system.libpath to true, then Oozie would > >only include the oozie-*.jar from the sharelib (the one with the Main) in > >it; if its set to true, then Oozie would include all of the jars in the > >sharelib. This would allow users to still use their own lib dir while > >still getting the Main into the classpath. > > > >- Robert > > > > > > > >On Tue, Apr 23, 2013 at 10:46 PM, Alejandro Abdelnur > ><tuc...@gmail.com>wrote: > > > >> virag, if your oozie servers dont have sharelibs currently installed > >>then > >> unstalling sharelibs as part of an update would not break ant running > >>app. > >> regarding the requirement of enabling the sharelibs to have the mains, > >>you > >> have a point. we need to figure out how to get those in the classpath if > >> the user is not using the sharelibs. i'd suggest we leave this > >> patch in trunk for now and we move it to a release branch once we sorted > >> out a way of solving this incompatibility. > >> > >> thanks > >> > >> Alejandro > >> (phone typing) > >> > >> On Apr 23, 2013, at 5:09 PM, Virag Kothari <vi...@yahoo-inc.com> wrote: > >> > >> > > >> > Tucu, Thanks for the reply and explanation. > >> > For workflows running in Y!, cold upgrade is not a feasible option as > >>we > >> > cannot expect all the running actions to complete before Oozie > >>restart. > >> > Hot upgrade is a good option, but we will need time to implement it > >>as we > >> > currently don't have sharelib as part of our deployment. > >> > So can we deprecate the refactoring of launcher classes in this > >>release? > >> > Also, all the launcher classes are only added to distributed cache if > >>the > >> > workflow is configured to use system lib path > >> > (oozie.use.system.libpath=true) > >> > We shouldn't mandate the workflow to have this property as they are > >> > oozie's launcher classes. > >> > > >> > Thanks, > >> > Virag > >> > > >> > > >> > > >> > On 4/23/13 3:21 PM, "Alejandro Abdelnur" <tuc...@gmail.com> wrote: > >> > > >> >> Virag, > >> >> > >> >> On sharelib being required, yes you are correct. For this we should: > >> >> > >> >> * Make 100% clear in the quick-start/install docs that the sharelib > >>is > >> >> REQUIRED. > >> >> > >> >> * Add a check at Oozie startup to verify the sharelib dir exists, > >>else > >> >> fail > >> >> to start. > >> >> > >> >> On sharelib lib issue during upgrade for in-flight jobs. Depending on > >> the > >> >> type of upgrade this may be an issue. > >> >> > >> >> * If you are upgrading an oozie server fix that does not change the > >> >> sharelib files, this is not an issue and you can just shutdown the > >>oozie > >> >> server with in-flight jobs. > >> >> > >> >> * If you are upgrading an oozie server fix that involves sharelib > >>files > >> >> changes, then you have 2 options: > >> >> > >> >> ** Cold upgrade: bring all WF jobs to suspend/completion, wait till > >>all > >> >> running actions end, then shutdown oozie server, upgrade oozie server > >> and > >> >> sharelib. Then restart oozie server and resume WF jobs. In this case > >>all > >> >> new WF actions will use the new sharelib. > >> >> > >> >> ** Hot upgrade: stop oozie server. modify the oozie-site.xml sharelib > >> >> location to point to a new directory. upgrade the oozie server. > >>install > >> >> the > >> >> sharelib (will be a create as the sharelib dir in HDFS does not > >>exist). > >> >> start the oozie server. In this case all running WF actions will > >> continue > >> >> running with no issues as the JARs in the distributed cache have not > >> been > >> >> touch. All new WF actions will start using the new sharelib. > >> >> > >> >> Note that this sharelib upgrade protocol is not introduced by > >>requiring > >> >> sharelib, it is required if you have applications that use sharelib > >> today. > >> >> > >> >> Does this address your concerns? > >> >> > >> >> Thanks. > >> >> > >> >> > >> >> > >> >> On Tue, Apr 23, 2013 at 1:35 PM, Virag Kothari <vi...@yahoo-inc.com> > >> >> wrote: > >> >> > >> >>> Hi, > >> >>> > >> >>> With OOZIE-1311 and its subtasks, the idea seems to move all the > >> >>> launcher > >> >>> classes like PigMain, HiveMain etc. to their respective sharelibs. > >> >>> So, now shared lib is a mandatory deployment step. Before shared lib > >> was > >> >>> optional as users could bundle jars with their workflow application. > >> >>> So always requiring shared lib seems to introduce 2 problems: > >> >>> > >> >>> 1. The current deployments which don't use action shared lib will > >> >>> fail. > >> >>> So, probably we should deprecate the current behavior. > >> >>> > >> >>> 2. The hadoop distributed cache mechanism will fail a job if the > >>files > >> >>> in > >> >>> DC are updated on hdfs while the hadoop job is running. So, when > >>Oozie > >> >>> is > >> >>> restarted and shared lib is uploaded to hdfs as part of > >> >>> deployment, hadoop will fail the existing jobs for > >>which > >> >>> the > >> >>> timestamp of the file on hdfs doesn't match the timestamp of its > >>copy > >> >>> in > >> >>> the job's DC. > >> >>> > >> >>> > >> >>> Thanks, > >> >>> Virag > >> > > >> > >
