I think Satish raises a very good point. I have been long uncomfortable with
the sharelib change that breaks backward compatibility. This is not really a
minor incompatibility, potentially, this is a product busting incompatibility
since no one can run any actions if directly upgrading from 4.0 or before to
the new version. And many of our users and customers will indeed upgrade oozie
once this is released. And many confusions will follow. I think we should bring
back the old format "/user/${user.name}/share/lib/" to co-exist with the new
format and make it backward compatible.
Any thoughts from others?
Bowen
On Wednesday, April 9, 2014 2:54 PM, Purshotam Shah <[email protected]>
wrote:
Bowen is correct. You need to upload using oozie-setup.sh sharelib create
to sharelib. It copies sharelib to a new timestamp directory. One can
upload multiple version of sharelib. Server will cleanup the older
sharelib. Earlier user have to clean it manually.
If you don¹t want to use oozie-setup.sh to setup sharelib, you can use
metafile configuration.
For Doc..
------------
Sharelib mapping file can be also configured. Configured file is a key
value mapping, where key will be the sharelib name for the
action and value is a comma separated list of DFS directories or jar files.
This can be configured in oozie-site.xml as :
<verbatim>
<!-- OOZIE -->
<property>
<name>oozie.service.ShareLibService.mapping.file</name>
<value></value>
<description>
Sharelib mapping files contains list of key=value,
where key will be the sharelib name for the action and value
is a comma separated list of
DFS directories or jar files.
Example.
oozie.pig_10=hdfs:///share/lib/pig/pig-0.10.1/lib/
oozie.pig=hdfs:///share/lib/pig/pig-0.11.1/lib/
oozie.distcp=hdfs:///share/lib/hadoop-2.2.0/share/hadoop/tools/lib/hadoop-d
istcp-2.2.0.jar
</description>
</property>
</verbatim>
------------
Puru.
On 4/9/14, 12:48 PM, "bowen zhang" <[email protected]> wrote:
>no, it's not. Before, user can manually upload sharelib to hdfs by using
>"hadoop dfs -put share share" command. Right now, you have to use the
>³uoozie-setup.sh sharelib create" command to upload sharelib. Puru from
>yahoo can address this issue.
>
>
>On Wednesday, April 9, 2014 12:10 PM, Satish Mittal
><[email protected]> wrote:
>
>Hi,
>
>I built oozie trunk and started oozie server with default config options.
>The oozie share lib directory is present in HDFS at usual location
>/user/oozie/share/lib. However oozie server logs contained:
>
>2014-04-09 18:17:57,507 WARN main ConfigurationService - SERVER[dev1]
>USER[-] GROUP[-] Configuration property
>[oozie.service.ShareLibService.mapping.file] not found, using default []
>2014-04-09 18:29:05,118 INFO main ShareLibService - SERVER[dev1] USER[-]
>GROUP[-] *No share lib directory found*
>2014-04-09 18:31:10,068 ERROR main ShareLibService - SERVER[dev1] USER[-]
>GROUP[-] Not able to cache shareLib. Admin need to issue oozlie cli
>command
>to update sharelib.
>java.lang.NullPointerException
> at
>org.apache.oozie.service.ShareLibService.init(ShareLibService.java:108)
> at
>org.apache.oozie.service.Services.setServiceInternal(Services.java:368)
> at org.apache.oozie.service.Services.setService(Services.java:354)
> at
>org.apache.oozie.service.Services.loadServices(Services.java:287)
> at org.apache.oozie.service.Services.init(Services.java:208)
> at
>org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.
>java:45)
> at
>org.apache.catalina.core.StandardContext.listenerStart(StandardContext.jav
>a:4206)
> at
>org.apache.catalina.core.StandardContext.start(StandardContext.java:4705)
> at
>org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java
>:799)
> at
>org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
> at
>org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
>at
>org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:67
>5)
> at
>org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:6
>01)
> at
>org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502)
> at
>org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
> at
>org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
> at
>org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupp
>ort.java:142)
> at
>org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
>at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
> at
>org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
>at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
> at
>org.apache.catalina.core.StandardService.start(StandardService.java:525)
>at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
> at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
>57)
> at
>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
>pl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
>2014-04-09 18:31:15,116 INFO main Services - SERVER[dev1] Initialized
>2014-04-09 18:31:15,124 INFO main Services - SERVER[dev1] Running with
>JARs for Hadoop version [0.20.2-cdh3u5]
>2014-04-09 18:31:15,125 INFO main Services - SERVER[dev1] Oozie System ID
>[oozie-oozi] started!
>
>Consequently workflow action launch submit failed since it couldn't find
>oozie share lib location:
>
>JOB[0000002-140409110726107-oozie-oozi-W]
>ACTION[0000002-140409110726107-oozie-oozi-W@eviction] Error starting
>action
>[eviction]. ErrorType [ERROR], ErrorCode [IndexOutOfBoundsException],
>Message [IndexOutOfBoundsException: Index: 0, Size: 0]
>org.apache.oozie.action.ActionExecutorException:
>IndexOutOfBoundsException:
>Index: 0, Size: 0
>at
>org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.jav
>a:401)
>at
>org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActio
>nExecutor.java:929)
>at
>org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor
>.java:1084)
>at
>org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXComman
>d.java:228)
>at
>org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXComman
>d.java:63)
>at org.apache.oozie.command.XCommand.call(XCommand.java:283)
>at
>org.apache.oozie.service.CallableQueueService$CompositeCallable.call(Calla
>bleQueueService.java:323)
>at
>org.apache.oozie.service.CallableQueueService$CompositeCallable.call(Calla
>bleQueueService.java:252)
>at
>org.apache.oozie.service.CallableQueueService$CallableWrapper.run(Callable
>QueueService.java:174)
>at
>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
>1110)
>at
>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java
>:603)
>at java.lang.Thread.run(Thread.java:679)
>Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
>at java.util.ArrayList.rangeCheck(ArrayList.java:571)
>at java.util.ArrayList.get(ArrayList.java:349)
>at
>org.apache.oozie.action.hadoop.JavaActionExecutor.addSystemShareLibForActi
>on(JavaActionExecutor.java:557)
>at
>org.apache.oozie.action.hadoop.JavaActionExecutor.addAllShareLibs(JavaActi
>onExecutor.java:649)
>at
>org.apache.oozie.action.hadoop.JavaActionExecutor.setLibFilesArchives(Java
>ActionExecutor.java:640)
>at
>org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActio
>nExecutor.java:834)
>... 10 more
>
>Does this mean that *current oozie trunk is not backward compatible with
>earlier oozie behavior*, where I could just deploy oozie share lib in HDFS
>at the system libpath provided in config, and oozie server would then find
>it and work with it?
>
>Regards,
>Satish
>
>--
>_____________________________________________________________
>The information contained in this communication is intended solely for
>the
>use of the individual or entity to whom it is addressed and others
>authorized to receive it. It may contain confidential or legally
>privileged
>information. If you are not the intended recipient you are hereby
>notified
>that any disclosure, copying, distribution or taking any action in
>reliance
>on the contents of this information is strictly prohibited and may be
>unlawful. If you have received this communication in error, please notify
>us immediately by responding to this email and then delete it from your
>system. The firm is neither liable for the proper and complete
>transmission
>of the information contained in this communication nor for any delay in
>its
>receipt.
