I agree that its generally a good idea to deprecate something when we change/remove it. That said, even though the sharelib was technically optional, don't we strongly recommend that users install it? Ideally, its a behind-the-scence change that users shouldn't really notice.
Though if we were to deprecate it, how would that work? I suppose we could have Oozie include the Main class in the launcher jar as it was doing before, but only when oozie.use.system.libpath is not true; I'm not sure if this would work, but to do that, we could have the assembly copy the oozie-*.jar from each sharelib into the libs dir so they're on the classpath (but still built/tested in separate modules). thanks - Robert On Tue, Apr 23, 2013 at 5:09 PM, Virag Kothari <vi...@yahoo-inc.com> wrote: > > Tucu, Thanks for the reply and explanation. > For workflows running in Y!, cold upgrade is not a feasible option as we > cannot expect all the running actions to complete before Oozie restart. > Hot upgrade is a good option, but we will need time to implement it as we > currently don't have sharelib as part of our deployment. > So can we deprecate the refactoring of launcher classes in this release? > Also, all the launcher classes are only added to distributed cache if the > workflow is configured to use system lib path > (oozie.use.system.libpath=true) > We shouldn't mandate the workflow to have this property as they are > oozie's launcher classes. > > Thanks, > Virag > > > > On 4/23/13 3:21 PM, "Alejandro Abdelnur" <tuc...@gmail.com> wrote: > > >Virag, > > > >On sharelib being required, yes you are correct. For this we should: > > > >* Make 100% clear in the quick-start/install docs that the sharelib is > >REQUIRED. > > > >* Add a check at Oozie startup to verify the sharelib dir exists, else > >fail > >to start. > > > >On sharelib lib issue during upgrade for in-flight jobs. Depending on the > >type of upgrade this may be an issue. > > > >* If you are upgrading an oozie server fix that does not change the > >sharelib files, this is not an issue and you can just shutdown the oozie > >server with in-flight jobs. > > > >* If you are upgrading an oozie server fix that involves sharelib files > >changes, then you have 2 options: > > > >** Cold upgrade: bring all WF jobs to suspend/completion, wait till all > >running actions end, then shutdown oozie server, upgrade oozie server and > >sharelib. Then restart oozie server and resume WF jobs. In this case all > >new WF actions will use the new sharelib. > > > >** Hot upgrade: stop oozie server. modify the oozie-site.xml sharelib > >location to point to a new directory. upgrade the oozie server. install > >the > >sharelib (will be a create as the sharelib dir in HDFS does not exist). > >start the oozie server. In this case all running WF actions will continue > >running with no issues as the JARs in the distributed cache have not been > >touch. All new WF actions will start using the new sharelib. > > > >Note that this sharelib upgrade protocol is not introduced by requiring > >sharelib, it is required if you have applications that use sharelib today. > > > >Does this address your concerns? > > > >Thanks. > > > > > > > >On Tue, Apr 23, 2013 at 1:35 PM, Virag Kothari <vi...@yahoo-inc.com> > >wrote: > > > >> Hi, > >> > >> With OOZIE-1311 and its subtasks, the idea seems to move all the > >>launcher > >> classes like PigMain, HiveMain etc. to their respective sharelibs. > >> So, now shared lib is a mandatory deployment step. Before shared lib was > >> optional as users could bundle jars with their workflow application. > >> So always requiring shared lib seems to introduce 2 problems: > >> > >> 1. The current deployments which don't use action shared lib will > >>fail. > >> So, probably we should deprecate the current behavior. > >> > >> 2. The hadoop distributed cache mechanism will fail a job if the files > >>in > >> DC are updated on hdfs while the hadoop job is running. So, when Oozie > >>is > >> restarted and shared lib is uploaded to hdfs as part of > >> deployment, hadoop will fail the existing jobs for which > >>the > >> timestamp of the file on hdfs doesn't match the timestamp of its copy > >>in > >> the job's DC. > >> > >> > >> Thanks, > >> Virag > >> > >> > >> > >> > >> > >
