Thanks Robert. Created OOZIE-1341 On 4/24/13 11:36 AM, "Robert Kanter" <rkan...@cloudera.com> wrote:
>I think I follow what you're saying; assuming everything works correctly, >I >think that sounds good. Can you make a JIRA with that and put up a patch; >that might be easier to see how this would work. > >thanks >- Robert > > >On Wed, Apr 24, 2013 at 11:29 AM, Virag Kothari <vi...@yahoo-inc.com> >wrote: > >> We can keep the Main classes in the share lib, but have the action >> executors package the main classes for shipping it to >> Launcher (basically again having the method getLauncherClasses() in >>action >> executors). Also the oozie-sharelib.jar containing main classes >> should be part of web-app. >> This will improve the building/testing env as classes are in isolated >> sharelibs but will make the share lib optional as main classes are part >>of >> oozie-server. >> Can we do this change to trunk as this is blocking our QE's to pick >>builds >> due to share lib incompatibility? >> >> We can still do what oozie-1318 is trying to achieve. For, e.g with Hcat >> integration, >> there can be multiple overridable implementation of URIHandlers that can >> be shipped to launcher. >> We can do the same for other main classes if required. >> >> >> Thanks, >> Virag >> >> >> On 4/24/13 10:18 AM, "Robert Kanter" <rkan...@cloudera.com> wrote: >> >> >That's true, its primarily a building/testing improvement. And from a >> >design perspective, it could be considered "cleaner" to not have the >> >launcher jar. >> > >> >Your idea sounds like a good compromise between backwards compatibility >> >and >> >improving the building/testing env. How would we do that? >> > >> >As a side note, there's also a new feature with the Main classes being >>in >> >the sharelib; users can now override the Main class that Oozie will use >> >for >> >the action because its no longer in the launcher jar. I think this >>could >> >still be a useful feature, but I'm guessing that if we do the >>compromise, >> >this would go away? In any case, OOZIE-1318 is to make the Main >> >overridable with a config, which is probably cleaner and more flexible >> >than >> >replacing a jar file. >> > >> >thanks >> >- Robert >> > >> > >> >On Wed, Apr 24, 2013 at 9:54 AM, Virag Kothari <vi...@yahoo-inc.com> >> >wrote: >> > >> >> That is a good suggestion for preserving compatibility, Robert. >> >> >> >> Another question is what is the main purpose of OOZIE-1311? From JIRA >> >> description, the primary >> >> advantage of moving Main classes from core to share lib is it can >>help >> >>in >> >> preventing dependencies conflicts >> >> (different versions of antlr's for pig, hive). However, that problem >> >>only >> >> exists while building/testing Oozie. >> >> If that is the case, main classes can be moved to share lib but they >>can >> >> be bundled in oozie-server itself. >> >> In that case, the deployment doesn't change and share lib can still >>be >> >> optional >> >> as oozie server would ship those classes (same as what was happening >> >> before). And, >> >> advantage of 1311 for building/testing will be retained >> >> >> >> The main concern is having share lib as a required installation will >> >> complicate deployment. >> >> Doing hot upgrade for share lib is non-trivial and Oozie doesn't >>provide >> >> inbuilt support for that. >> >> Even though end-users wouldn't notice the change, it will be a big >> >>change >> >> for system admins installing oozie >> >> So even though share lib is nice feature to have, keeping it optional >> >> seems better to me as there are other ways >> >> Of including jars and which were introduced much before share lib. >> >> >> >> Thanks, >> >> Virag >> >> >> >> >> >> >> >> >> >> On 4/24/13 9:09 AM, "Robert Kanter" <rkan...@cloudera.com> wrote: >> >> >> >> >Alejandro had another suggestion to prevent breaking stuff for >>existing >> >> >apps where the sharelib isn't used. Assuming the sharelib is >> >>installed, >> >> >if >> >> >the user doesn't set oozie.use.system.libpath to true, then Oozie >>would >> >> >only include the oozie-*.jar from the sharelib (the one with the >>Main) >> >>in >> >> >it; if its set to true, then Oozie would include all of the jars in >>the >> >> >sharelib. This would allow users to still use their own lib dir >>while >> >> >still getting the Main into the classpath. >> >> > >> >> >- Robert >> >> > >> >> > >> >> > >> >> >On Tue, Apr 23, 2013 at 10:46 PM, Alejandro Abdelnur >> >> ><tuc...@gmail.com>wrote: >> >> > >> >> >> virag, if your oozie servers dont have sharelibs currently >>installed >> >> >>then >> >> >> unstalling sharelibs as part of an update would not break ant >>running >> >> >>app. >> >> >> regarding the requirement of enabling the sharelibs to have the >> >>mains, >> >> >>you >> >> >> have a point. we need to figure out how to get those in the >> >>classpath if >> >> >> the user is not using the sharelibs. i'd suggest we leave this >> >> >> patch in trunk for now and we move it to a release branch once we >> >>sorted >> >> >> out a way of solving this incompatibility. >> >> >> >> >> >> thanks >> >> >> >> >> >> Alejandro >> >> >> (phone typing) >> >> >> >> >> >> On Apr 23, 2013, at 5:09 PM, Virag Kothari <vi...@yahoo-inc.com> >> >>wrote: >> >> >> >> >> >> > >> >> >> > Tucu, Thanks for the reply and explanation. >> >> >> > For workflows running in Y!, cold upgrade is not a feasible >>option >> >>as >> >> >>we >> >> >> > cannot expect all the running actions to complete before Oozie >> >> >>restart. >> >> >> > Hot upgrade is a good option, but we will need time to >>implement it >> >> >>as we >> >> >> > currently don't have sharelib as part of our deployment. >> >> >> > So can we deprecate the refactoring of launcher classes in this >> >> >>release? >> >> >> > Also, all the launcher classes are only added to distributed >>cache >> >>if >> >> >>the >> >> >> > workflow is configured to use system lib path >> >> >> > (oozie.use.system.libpath=true) >> >> >> > We shouldn't mandate the workflow to have this property as they >>are >> >> >> > oozie's launcher classes. >> >> >> > >> >> >> > Thanks, >> >> >> > Virag >> >> >> > >> >> >> > >> >> >> > >> >> >> > On 4/23/13 3:21 PM, "Alejandro Abdelnur" <tuc...@gmail.com> >>wrote: >> >> >> > >> >> >> >> Virag, >> >> >> >> >> >> >> >> On sharelib being required, yes you are correct. For this we >> >>should: >> >> >> >> >> >> >> >> * Make 100% clear in the quick-start/install docs that the >> >>sharelib >> >> >>is >> >> >> >> REQUIRED. >> >> >> >> >> >> >> >> * Add a check at Oozie startup to verify the sharelib dir >>exists, >> >> >>else >> >> >> >> fail >> >> >> >> to start. >> >> >> >> >> >> >> >> On sharelib lib issue during upgrade for in-flight jobs. >> >>Depending on >> >> >> the >> >> >> >> type of upgrade this may be an issue. >> >> >> >> >> >> >> >> * If you are upgrading an oozie server fix that does not change >> >>the >> >> >> >> sharelib files, this is not an issue and you can just shutdown >>the >> >> >>oozie >> >> >> >> server with in-flight jobs. >> >> >> >> >> >> >> >> * If you are upgrading an oozie server fix that involves >>sharelib >> >> >>files >> >> >> >> changes, then you have 2 options: >> >> >> >> >> >> >> >> ** Cold upgrade: bring all WF jobs to suspend/completion, wait >> >>till >> >> >>all >> >> >> >> running actions end, then shutdown oozie server, upgrade oozie >> >>server >> >> >> and >> >> >> >> sharelib. Then restart oozie server and resume WF jobs. In this >> >>case >> >> >>all >> >> >> >> new WF actions will use the new sharelib. >> >> >> >> >> >> >> >> ** Hot upgrade: stop oozie server. modify the oozie-site.xml >> >>sharelib >> >> >> >> location to point to a new directory. upgrade the oozie server. >> >> >>install >> >> >> >> the >> >> >> >> sharelib (will be a create as the sharelib dir in HDFS does not >> >> >>exist). >> >> >> >> start the oozie server. In this case all running WF actions >>will >> >> >> continue >> >> >> >> running with no issues as the JARs in the distributed cache >>have >> >>not >> >> >> been >> >> >> >> touch. All new WF actions will start using the new sharelib. >> >> >> >> >> >> >> >> Note that this sharelib upgrade protocol is not introduced by >> >> >>requiring >> >> >> >> sharelib, it is required if you have applications that use >> >>sharelib >> >> >> today. >> >> >> >> >> >> >> >> Does this address your concerns? >> >> >> >> >> >> >> >> Thanks. >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Tue, Apr 23, 2013 at 1:35 PM, Virag Kothari >> >><vi...@yahoo-inc.com> >> >> >> >> wrote: >> >> >> >> >> >> >> >>> Hi, >> >> >> >>> >> >> >> >>> With OOZIE-1311 and its subtasks, the idea seems to move all >>the >> >> >> >>> launcher >> >> >> >>> classes like PigMain, HiveMain etc. to their respective >> >>sharelibs. >> >> >> >>> So, now shared lib is a mandatory deployment step. Before >>shared >> >>lib >> >> >> was >> >> >> >>> optional as users could bundle jars with their workflow >> >>application. >> >> >> >>> So always requiring shared lib seems to introduce 2 problems: >> >> >> >>> >> >> >> >>> 1. The current deployments which don't use action shared lib >> >>will >> >> >> >>> fail. >> >> >> >>> So, probably we should deprecate the current behavior. >> >> >> >>> >> >> >> >>> 2. The hadoop distributed cache mechanism will fail a job if >>the >> >> >>files >> >> >> >>> in >> >> >> >>> DC are updated on hdfs while the hadoop job is running. So, >>when >> >> >>Oozie >> >> >> >>> is >> >> >> >>> restarted and shared lib is uploaded to hdfs as part of >> >> >> >>> deployment, hadoop will fail the existing jobs >>for >> >> >>which >> >> >> >>> the >> >> >> >>> timestamp of the file on hdfs doesn't match the timestamp of >>its >> >> >>copy >> >> >> >>> in >> >> >> >>> the job's DC. >> >> >> >>> >> >> >> >>> >> >> >> >>> Thanks, >> >> >> >>> Virag >> >> >> > >> >> >> >> >> >> >> >> >>
