Thanks Puru and Bowen. I have created OOZIE-1789 to support backward compatibility of share lib.
-Satish On Thu, Apr 10, 2014 at 10:38 AM, Purshotam Shah <[email protected]>wrote: > +1 > If no new format is found we can fallback to old format. > > If both old and new format are present, new will be used and old will be > eligible for purging. Last modified time of old sharelib directory can be > use for purging logic. > > Puru. > > Sent from my iPhone > > On Apr 9, 2014, at 9:49 PM, "bowen zhang" <[email protected]<mailto: > [email protected]>> wrote: > > I think Satish raises a very good point. I have been long uncomfortable > with the sharelib change that breaks backward compatibility. This is not > really a minor incompatibility, potentially, this is a product busting > incompatibility since no one can run any actions if directly upgrading from > 4.0 or before to the new version. And many of our users and customers will > indeed upgrade oozie once this is released. And many confusions will > follow. I think we should bring back the old format > "/user/${user.name}/share/lib/" > to co-exist with the new format and make it backward compatible. > Any thoughts from others? > Bowen > > On Wednesday, April 9, 2014 2:54 PM, Purshotam Shah < > [email protected]<mailto:[email protected]>> wrote: > Bowen is correct. You need to upload using oozie-setup.sh sharelib create > to sharelib. It copies sharelib to a new timestamp directory. One can > upload multiple version of sharelib. Server will cleanup the older > sharelib. Earlier user have to clean it manually. > > If you don¹t want to use oozie-setup.sh to setup sharelib, you can use > metafile configuration. > > For Doc.. > ------------ > Sharelib mapping file can be also configured. Configured file is a key > value mapping, where key will be the sharelib name for the > action and value is a comma separated list of DFS directories or jar files. > This can be configured in oozie-site.xml as : > <verbatim> > <!-- OOZIE --> > <property> > <name>oozie.service.ShareLibService.mapping.file</name> > <value></value> > <description> > Sharelib mapping files contains list of key=value, > where key will be the sharelib name for the action and value > is a comma separated list of > DFS directories or jar files. > Example. > oozie.pig_10=hdfs:///share/lib/pig/pig-0.10.1/lib/ > oozie.pig=hdfs:///share/lib/pig/pig-0.11.1/lib/ > > oozie.distcp=hdfs:///share/lib/hadoop-2.2.0/share/hadoop/tools/lib/hadoop-d > istcp-2.2.0.jar > </description> > </property> > </verbatim> > ------------ > > > Puru. > > > On 4/9/14, 12:48 PM, "bowen zhang" <[email protected]<mailto: > [email protected]>> wrote: > > >no, it's not. Before, user can manually upload sharelib to hdfs by using > >"hadoop dfs -put share share" command. Right now, you have to use the > >³uoozie-setup.sh sharelib create" command to upload sharelib. Puru from > > >yahoo can address this issue. > > > > > >On Wednesday, April 9, 2014 12:10 PM, Satish Mittal > ><[email protected]<mailto:[email protected]>> wrote: > > > >Hi, > > > >I built oozie trunk and started oozie server with default config options. > >The oozie share lib directory is present in HDFS at usual location > >/user/oozie/share/lib. However oozie server logs contained: > > > >2014-04-09 18:17:57,507 WARN main ConfigurationService - SERVER[dev1] > >USER[-] GROUP[-] Configuration property > >[oozie.service.ShareLibService.mapping.file] not found, using default [] > >2014-04-09 18:29:05,118 INFO main ShareLibService - SERVER[dev1] USER[-] > >GROUP[-] *No share lib directory found* > >2014-04-09 18:31:10,068 ERROR main ShareLibService - SERVER[dev1] USER[-] > >GROUP[-] Not able to cache shareLib. Admin need to issue oozlie cli > >command > >to update sharelib. > >java.lang.NullPointerException > > at > >org.apache.oozie.service.ShareLibService.init(ShareLibService.java:108) > > at > >org.apache.oozie.service.Services.setServiceInternal(Services.java:368) > > at org.apache.oozie.service.Services.setService(Services.java:354) > > at > >org.apache.oozie.service.Services.loadServices(Services.java:287) > > at org.apache.oozie.service.Services.init(Services.java:208) > > at > >org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader. > >java:45) > > at > >org.apache.catalina.core.StandardContext.listenerStart(StandardContext.jav > >a:4206) > > at > >org.apache.catalina.core.StandardContext.start(StandardContext.java:4705) > > at > >org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java > >:799) > > at > >org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779) > > at > >org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601) > >at > >org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:67 > >5) > > at > >org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:6 > >01) > > at > >org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502) > > at > >org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317) > > at > >org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324) > > at > >org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupp > >ort.java:142) > > at > >org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065) > >at org.apache.catalina.core.StandardHost.start(StandardHost.java:840) > > at > >org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057) > >at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463) > > at > >org.apache.catalina.core.StandardService.start(StandardService.java:525) > >at org.apache.catalina.core.StandardServer.start(StandardServer.java:754) > > at org.apache.catalina.startup.Catalina.start(Catalina.java:595) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > >sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: > >57) > > at > >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm > >pl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:616) > > at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289) > > at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414) > >2014-04-09 18:31:15,116 INFO main Services - SERVER[dev1] Initialized > >2014-04-09 18:31:15,124 INFO main Services - SERVER[dev1] Running with > >JARs for Hadoop version [0.20.2-cdh3u5] > >2014-04-09 18:31:15,125 INFO main Services - SERVER[dev1] Oozie System ID > >[oozie-oozi] started! > > > >Consequently workflow action launch submit failed since it couldn't find > >oozie share lib location: > > > >JOB[0000002-140409110726107-oozie-oozi-W] > >ACTION[0000002-140409110726107-oozie-oozi-W@eviction<mailto: > 0000002-140409110726107-oozie-oozi-W@eviction>] Error starting > >action > >[eviction]. ErrorType [ERROR], ErrorCode [IndexOutOfBoundsException], > >Message [IndexOutOfBoundsException: Index: 0, Size: 0] > >org.apache.oozie.action.ActionExecutorException: > >IndexOutOfBoundsException: > >Index: 0, Size: 0 > >at > >org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.jav > >a:401) > >at > >org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActio > >nExecutor.java:929) > >at > >org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor > >.java:1084) > >at > >org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXComman > >d.java:228) > >at > >org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXComman > >d.java:63) > >at org.apache.oozie.command.XCommand.call(XCommand.java:283) > >at > >org.apache.oozie.service.CallableQueueService$CompositeCallable.call(Calla > >bleQueueService.java:323) > >at > >org.apache.oozie.service.CallableQueueService$CompositeCallable.call(Calla > >bleQueueService.java:252) > >at > >org.apache.oozie.service.CallableQueueService$CallableWrapper.run(Callable > >QueueService.java:174) > >at > >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: > >1110) > >at > >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java > >:603) > >at java.lang.Thread.run(Thread.java:679) > >Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0 > >at java.util.ArrayList.rangeCheck(ArrayList.java:571) > >at java.util.ArrayList.get(ArrayList.java:349) > >at > >org.apache.oozie.action.hadoop.JavaActionExecutor.addSystemShareLibForActi > >on(JavaActionExecutor.java:557) > >at > >org.apache.oozie.action.hadoop.JavaActionExecutor.addAllShareLibs(JavaActi > >onExecutor.java:649) > >at > >org.apache.oozie.action.hadoop.JavaActionExecutor.setLibFilesArchives(Java > >ActionExecutor.java:640) > >at > >org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActio > >nExecutor.java:834) > >... 10 more > > > >Does this mean that *current oozie trunk is not backward compatible with > >earlier oozie behavior*, where I could just deploy oozie share lib in HDFS > >at the system libpath provided in config, and oozie server would then find > >it and work with it? > > > >Regards, > >Satish > > > >-- > >_____________________________________________________________ > >The information contained in this communication is intended solely for > >the > >use of the individual or entity to whom it is addressed and others > >authorized to receive it. It may contain confidential or legally > >privileged > >information. If you are not the intended recipient you are hereby > >notified > >that any disclosure, copying, distribution or taking any action in > >reliance > >on the contents of this information is strictly prohibited and may be > >unlawful. If you have received this communication in error, please notify > >us immediately by responding to this email and then delete it from your > >system. The firm is neither liable for the proper and complete > >transmission > >of the information contained in this communication nor for any delay in > >its > >receipt. > > > -- _____________________________________________________________ The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system. The firm is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.
