Dear oozie user and dev,
I set following values in job.properties and in action properties in CDH
5.5.0 Hue oozie:
oozie.use.system.libpath=true
oozie.libpath=hdfs://ip-10-0-4-248.us-west-1.compute.internal:8020/user/oozie/share/lib/lib_20151201085935
oozie.action.sharelib.for.spark=hdfs://ip-10-0-4-248.us-west-1.compute.internal:8020/user/oozie/share/lib/lib_20151201085935/spark
*Here is the workflow file:*
<workflow-app name="sparktest-cassandra" xmlns="uri:oozie:workflow:0.5">
<start to="spark-b23b"/>
<kill name="Kill">
<message>Action failed, error
message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<action name="spark-b23b">
<spark xmlns="uri:oozie:spark-action:0.1">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>spark.executor.extraClassPath</name>
<value>lib/guava-16.0.1.jar</value>
</property>
<property>
<name>oozie.action.sharelib.for.spark</name>
<value>/user/oozie/share/lib/lib_20151201085935/spark</value>
</property>
<property>
<name>oozie.use.system.libpath</name>
<value>true</value>
</property>
<property>
<name>oozie.libpath</name>
<value>/user/oozie/share/lib/lib_20151201085935</value>
</property>
</configuration>
<master>local[4]</master>
<mode>client</mode>
<name>sparktest-cassandra</name>
<class>TestCassandra</class>
<jar>lib/sparktest.jar</jar>
<spark-opts>--driver-class-path
/opt/cloudera/parcels/CDH/jars/guava-16.0.1.jar</spark-opts>
<arg>s3n://gridx-output/sparktest/ </arg>
<arg>10</arg>
<arg>3</arg>
<arg>2</arg>
</spark>
<ok to="End"/>
<error to="Kill"/>
</action>
<end name="End"/>
</workflow-app>
*job.properties file:*
oozie.use.system.libpath=true
security_enabled=False
oozie.libpath=hdfs://ip-10-0-4-248.us-west-1.compute.internal:8020/user/oozie/share/lib/lib_20151201085935
oozie.action.sharelib.for.spark=hdfs://ip-10-0-4-248.us-west-1.compute.internal:8020/user/oozie/share/lib/lib_20151201085935/spark
dryrun=False
jobTracker=ip-10-0-4-248.us-west-1.compute.internal:8032
nameNode=hdfs://ip-10-0-4-248.us-west-1.compute.internal:8020
But I got following SparkMain class not found issue:
<<< Invocation of Main class completed <<<
Failing Oozie Launcher, Main class
[org.apache.oozie.action.hadoop.SparkMain], exception invoking main(),
java.lang.ClassNotFoundException: Class
org.apache.oozie.action.hadoop.SparkMain not found
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
org.apache.oozie.action.hadoop.SparkMain not found
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2199)
at
org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:234)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:378)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:296)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: Class
org.apache.oozie.action.hadoop.SparkMain not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2105)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2197)
... 13 more
Oozie Launcher failed, finishing Hadoop job gracefully
Oozie Launcher, uploading action data to HDFS sequence file:
hdfs://ip-10-0-4-248.us-west-1.compute.internal:8020/user/admin/oozie-oozi/0000006-160224085347053-oozie-oozi-W/spark-b23b--spark/action-data.seq
<http://ip-10-0-4-248.us-west-1.compute.internal:8888/filebrowser/view=/user/admin/oozie-oozi/0000006-160224085347053-oozie-oozi-W/spark-b23b--spark/action-data.seq>
Oozie Launcher ends
Can you help to give any suggestion? Thanks a lot!
--
Cheers,
-----
Big Data - Big Wisdom - Big Value
--------------
Michelle Zhang (张莉苹)
