Hi:
I tried to use oozie spark action.
I am using spark 2.0
So I replace the spark shared lib with those jars under spark 2.0 /jars
I kept the oozie-sharelib-spark-4.2.0.jar
When I submit the action
I got following error:
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain],
main() threw exception, Library directory
'/data/tmp/hadoop-dbadmin/nm-local-dir/usercache/dbadmin/appcache/application_1478518257576_0002/container_1478518257576_0002_01_000001/assembly/target/scala-2.11/jars'
does not exist; make sure Spark is built.
java.lang.IllegalStateException: Library directory
'/data/tmp/hadoop-dbadmin/nm-local-dir/usercache/dbadmin/appcache/application_1478518257576_0002/container_1478518257576_0002_01_000001/assembly/target/scala-2.11/jars'
does not exist; make sure Spark is built.
at
org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:248)
at
org.apache.spark.launcher.CommandBuilderUtils.findJarsDir(CommandBuilderUtils.java:368)
at
org.apache.spark.launcher.YarnCommandBuilderUtils$.findJarsDir(YarnCommandBuilderUtils.scala:38)
at
org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:500)
at
org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:834)
at
org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:167)
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1119)
at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1175)
at org.apache.spark.deploy.yarn.Client.main(Client.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:729)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
at org.apache.oozie.action.hadoop.SparkMain.runSpark(SparkMain.java:104)
at org.apache.oozie.action.hadoop.SparkMain.run(SparkMain.java:95)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47)
at org.apache.oozie.action.hadoop.SparkMain.main(SparkMain.java:38)
help is appreciated.
Cheng Yi
