Hi Liping, If you change jars, or anything with the sharelib directories, you have to either restart the Oozie server or run the 'oozie admin -sharelibupdate' command for it to notice.
That said, CDH Spark requires Guava 14.0.1, which is what's in the Spark sharelib directory already. If you simply put Guava 16.0.1 in there, you may run into other problems. - Robert On Wed, Feb 24, 2016 at 12:34 PM, Liping Zhang <[email protected]> wrote: > I added guava-16.0.1.jar into > HDFS /user/oozie/share/lib/lib_20151201085935/spark dir, and chown as > "oozie:oozie", chmod 777, but it still could not find the jar. > > job.properties: > > oozie.use.system.libpath=True > security_enabled=False > dryrun=False > jobTracker=ip-10-0-4-248.us-west-1.compute.internal:8032 > nameNode=hdfs://ip-10-0-4-248.us-west-1.compute.internal:8020 > > > workflow.xml: > <workflow-app name="sparktest-cassandra" xmlns="uri:oozie:workflow:0.5"> > <start to="spark-b23b"/> > <kill name="Kill"> > <message>Action failed, error > message[${wf:errorMessage(wf:lastErrorNode())}]</message> > </kill> > <action name="spark-b23b"> > <spark xmlns="uri:oozie:spark-action:0.1"> > <job-tracker>${jobTracker}</job-tracker> > <name-node>${nameNode}</name-node> > <master>local[4]</master> > <mode>client</mode> > <name>sparktest-cassandra</name> > <class>TestCassandra</class> > <jar>lib/sparktest.jar</jar> > <spark-opts>--driver-class-path > /opt/cloudera/parcels/CDH/jars/guava-16.0.1.jar --jars > lib/*.jar</spark-opts> > <arg>s3n://gridx-output/sparktest/ </arg> > <arg>10</arg> > <arg>3</arg> > <arg>2</arg> > </spark> > <ok to="End"/> > <error to="Kill"/> > </action> > <end name="End"/> > </workflow-app> > > Appreciated if any help! > > On Wed, Feb 24, 2016 at 12:34 PM, Liping Zhang <[email protected]> > wrote: > > > Dear, > > > > We used CDH5.5.0 Hue Oozie to run Spark action, and in the spark job > > action, the job used spark-cassandra-connector_2.10-1.5.0-M2.jar. > > > > The job can run successfully with spark-submit command, but it was failed > > to run in oozie. > > > > It said oozie can not find guava-16.0.1.jar(guava-16.0.1.jar is an > > dependency of DSE4.8.3 Cassandra) as following exception. > > > > Do you know how to add external guava-16.0.1.jar in oozie class path? > > > > Thanks! > > ----- > > > > >>> Invoking Spark class now >>> > > > > > > <<< Invocation of Main class completed <<< > > > > Failing Oozie Launcher, Main class > > [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, > > com.google.common.reflect.TypeToken.isPrimitive()Z > > java.lang.NoSuchMethodError: > > com.google.common.reflect.TypeToken.isPrimitive()Z > > at com.datastax.driver.core.TypeCodec.<init>(TypeCodec.java:142) > > at com.datastax.driver.core.TypeCodec.<init>(TypeCodec.java:136) > > at > com.datastax.driver.core.TypeCodec$BlobCodec.<init>(TypeCodec.java:609) > > at > > com.datastax.driver.core.TypeCodec$BlobCodec.<clinit>(TypeCodec.java:606) > > at > com.datastax.driver.core.CodecRegistry.<clinit>(CodecRegistry.java:147) > > at > > > com.datastax.driver.core.Configuration$Builder.build(Configuration.java:259) > > at > > > com.datastax.driver.core.Cluster$Builder.getConfiguration(Cluster.java:1135) > > at com.datastax.driver.core.Cluster.<init>(Cluster.java:111) > > at com.datastax.driver.core.Cluster.buildFrom(Cluster.java:178) > > at com.datastax.driver.core.Cluster$Builder.build(Cluster.java:1152) > > at > > > com.datastax.spark.connector.cql.DefaultConnectionFactory$.createCluster(CassandraConnectionFactory.scala:85) > > at > > > com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:155) > > at > > > com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:150) > > at > > > com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:150) > > at > > > com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:31) > > at > > > com.datastax.spark.connector.cql.RefCountedCache.acquire(RefCountedCache.scala:56) > > at > > > com.datastax.spark.connector.cql.CassandraConnector.openSession(CassandraConnector.scala:81) > > at > > > com.datastax.spark.connector.cql.CassandraConnector.withSessionDo(CassandraConnector.scala:109) > > at > > > com.datastax.spark.connector.cql.CassandraConnector.withClusterDo(CassandraConnector.scala:120) > > at > com.datastax.spark.connector.cql.Schema$.fromCassandra(Schema.scala:241) > > at > > > com.datastax.spark.connector.writer.TableWriter$.apply(TableWriter.scala:263) > > at > > > com.datastax.spark.connector.RDDFunctions.saveToCassandra(RDDFunctions.scala:36) > > at TestCassandra$.main(TestCassandra.scala:44) > > at TestCassandra.main(TestCassandra.scala) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > ----- > > > > -- > > Cheers, > > ----- > > Big Data - Big Wisdom - Big Value > > -------------- > > Michelle Zhang (张莉苹) > > > > > > -- > Cheers, > ----- > Big Data - Big Wisdom - Big Value > -------------- > Michelle Zhang (张莉苹) >
