Could this be related to https://issues.apache.org/jira/browse/MAPREDUCE-121 ?
I do see that mapred.job.classpath.files is colon-delimited (with ':') but Oozie puts fully qualified HDFS paths in there (which contains colons in hdfs://localhost:9000/path). I can confirm that only the paths that don't have hdfs://localhost:9000/... at the beginning are correctly seen in the launcher's classpath. I'll try to change JavaActionExecutor to make sure things added to the distributed cache are not added through fully qualified URLs and see if that works. /Max -- Maxime Petazzoni Sr. Platform Engineer m 408.310.0595 www.turn.com ________________________________________ From: Maxime Petazzoni [[email protected]] Sent: Tuesday, August 06, 2013 2:31 PM To: [email protected] Subject: RE: Classpath (and extra JARs) for Java actions The Hadoop job config correctly shows all my JARs both in mapred.job.classpath.files and mapred.cache.files, which indicates that Oozie knows they are there and did something with it. Yet the classpath listed by the launcher doesn't show these JARs and my job fails. It seems cleaner to me to use the shared libs, and I have them deployed on HDFS and I see them in the listed classpath (apparently through distcache, if I read the path correctly). But I don't see any of the application's lib JAR files on the classpath. Any idea what's up? /Max -- Maxime Petazzoni Sr. Platform Engineer m 408.310.0595 www.turn.com ________________________________________ From: Virag Kothari [[email protected]] Sent: Tuesday, August 06, 2013 2:20 PM To: [email protected] Subject: Re: Classpath (and extra JARs) for Java actions Hi Max, Jars in application lib should be available for all actions. The documentation might be incorrect. Did you check your hadoop job config to see which jars are added to classpath? Following is the precedence order: 1) Application lib 2) oozie.libpath 3) oozie.use.system.libpath. Even though the priority is defined, it is recommended to use only one of the ways at a time. Also the switch to no launcher jar is not mandatory and governed by 'oozie.action.ship.launcher.jar' which is set to true for 4.x. So you are not forced (although recommended) to use shared lib. Thanks, Virag On 8/6/13 2:01 PM, "Maxime Petazzoni" <[email protected]> wrote: >Hi all, > >My Java action needs some extra JAR files. If I understand the >documentation (and my testing) correctly, the JARs I placed in the lib/ >folder in my application directory on HDFS are only added to the >classpath of MapReduce and Pig actions (why not all??). > >In the past I used oozie.libpath and that worked pretty well, but now in >Oozie 4.x with the switch to no launcher jar and the need for the >sharelibs to be on HDFS, I set oozie.use.system.libpath, which >apparently doesn't play well when oozie.libpath is also set >(JARs from oozie.libpath seems to be ignored, but interestingly not >other file types like text/config files?). > >What's the recommended way of having extra JARs for Java actions with >Oozie? What combination of oozie.libpath, oozie.use.system.libpath >should I use? > >Any help greatly appreciated! > >Thanks in advance, >/Max >-- >Maxime Petazzoni >Sr. Platform Engineer >m 408.310.0595 >www.turn.com
