Dong Ying, I have not changed anything. I have built oozie from source with - mvn clean install assembly:single -DskipTests -Dhadoop.version-2.6.0 -Puber
Note: -Puber is used so that dependencies of the Oozie server are copied Then - cd distro/target/oozie-5.0.0-SNAPSHOT - I made sure that my hadoop configuration (e.g. fs.defaultFS points to the right namenode in core-site.xml in conf/hadoop-conf ) was correct (see https://oozie.apache.org/docs/4.2.0/AG_HadoopConfiguration.html) - Started Oozie server bin/oozied.sh start - Installed Sharelib bin/oozie-setup.sh sharelib create -fs hdfs://localhost:9000 -locallib oozie-sharelib-5.0.0-SNAPSHOT.tar.gz -concurrency 4 - Updated Sharelib bin/oozie admin -sharelibupdate - Extracted, uploaded examples to HDFS - Submitted map-reduce example bin/oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/job.properties -run -DnameNode=hdfs://localhost:9000 -DjobTracker=localhost:8032 (I am working with a pseudo, single-node, Hadoop 2.6 cluster in my dev environment) JavaActionExecutor should add MR to classpath in case of a MapReduce action: https://github.com/apache/oozie/blob/83d4ddf45aa16649bd9fae367fa915379d5781cd/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L1091 In my debugger I could see the following env was added to launcherJobConf: {CLASSPATH=$PWD:$PWD/*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*} It is later added to appContext used for submitting the application to YARN - https://github.com/apache/oozie/blob/83d4ddf45aa16649bd9fae367fa915379d5781cd/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L1028 On Wed, Jun 14, 2017 at 5:39 AM, Dongying Jiao <[email protected]> wrote: > > Hi Attilia: > I have Oozie share lib, it is setup during oozie installation, but oozie > share lib doesn't have map-reduce folder. Did you modify anything to make > the example map-reduce workflow run successfully? > > Peter: > About the commit link you mentioned, I see MR jars are added from > map-reduce property:“mapreduce.application.classpath”, the classpath in my > cluster is: > /etc/hadoop/conf/:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/etc/hadoop/conf/secure:/usr/lib/hadoop-lzo/lib/*:/usr/iop/current/ext/hadoop/* > > Not sure if my map-reduce classpath is not corrcet. > I used the 08/Jun build in oozie master branch. > > Thanks for help. > > Best Regards, > Dong Ying > > > > 2017-06-13 23:27 GMT+08:00 Attila Sasvari (JIRA) <[email protected]>: > > > > > [ https://issues.apache.org/jira/browse/OOZIE-2941?page= > > com.atlassian.jira.plugin.system.issuetabpanels:comment- > > tabpanel&focusedCommentId=16048022#comment-16048022 ] > > > > Attila Sasvari commented on OOZIE-2941: > > --------------------------------------- > > > > [~Dongying Jiao] I re-tested the mapreduce example in my local > > environment, and the workflow successfully finished. Before running the MR > > workflow, you should first install Oozie sharelib and make sure jhs is > > running. As per your previous commend, I suspect sharelib was not installed > > in your case. So I believe this issue is not the same as my experience with > > the shell action example. > > > > > Old map-reduce workflow example can't work with OYA > > > --------------------------------------------------- > > > > > > Key: OOZIE-2941 > > > URL: https://issues.apache.org/jira/browse/OOZIE-2941 > > > Project: Oozie > > > Issue Type: Sub-task > > > Components: workflow > > > Affects Versions: trunk > > > Reporter: Dongying Jiao > > > Attachments: RM_UI.jpg > > > > > > > > > map-reduce workflow in Oozie examples run failed after OYA merged. The > > exception in yarn log is: > > > Exception in thread "main" java.lang.NoClassDefFoundError: > > org/apache/hadoop/mapred/Mapper > > > at java.lang.ClassLoader.defineClass1(Native Method) > > > at java.lang.ClassLoader.defineClass(ClassLoader.java:763) > > > at java.security.SecureClassLoader.defineClass( > > SecureClassLoader.java:142) > > > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) > > > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:368) > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > > > at java.security.AccessController.doPrivileged(Native Method) > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > > > at org.apache.oozie.action.hadoop.LauncherAM. > > getMainArguments(LauncherAM.java:540) > > > at org.apache.oozie.action.hadoop.LauncherAM.run( > > LauncherAM.java:170) > > > at org.apache.oozie.action.hadoop.LauncherAM.main( > > LauncherAM.java:148) > > > Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred. > > Mapper > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > > > ... 15 more > > > > > > > > -- > > This message was sent by Atlassian JIRA > > (v6.4.14#64029) > >
