I added guava-16.0.1.jar into
HDFS /user/oozie/share/lib/lib_20151201085935/spark dir, and chown as
"oozie:oozie", chmod 777, but it still could not find the jar.
job.properties:
oozie.use.system.libpath=True
security_enabled=False
dryrun=False
jobTracker=ip-10-0-4-248.us-west-1.compute.internal:8032
nameNode=hdfs://ip-10-0-4-248.us-west-1.compute.internal:8020
workflow.xml:
<workflow-app name="sparktest-cassandra" xmlns="uri:oozie:workflow:0.5">
<start to="spark-b23b"/>
<kill name="Kill">
<message>Action failed, error
message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<action name="spark-b23b">
<spark xmlns="uri:oozie:spark-action:0.1">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<master>local[4]</master>
<mode>client</mode>
<name>sparktest-cassandra</name>
<class>TestCassandra</class>
<jar>lib/sparktest.jar</jar>
<spark-opts>--driver-class-path
/opt/cloudera/parcels/CDH/jars/guava-16.0.1.jar --jars
lib/*.jar</spark-opts>
<arg>s3n://gridx-output/sparktest/ </arg>
<arg>10</arg>
<arg>3</arg>
<arg>2</arg>
</spark>
<ok to="End"/>
<error to="Kill"/>
</action>
<end name="End"/>
</workflow-app>
Appreciated if any help!
On Wed, Feb 24, 2016 at 12:34 PM, Liping Zhang <[email protected]>
wrote:
> Dear,
>
> We used CDH5.5.0 Hue Oozie to run Spark action, and in the spark job
> action, the job used spark-cassandra-connector_2.10-1.5.0-M2.jar.
>
> The job can run successfully with spark-submit command, but it was failed
> to run in oozie.
>
> It said oozie can not find guava-16.0.1.jar(guava-16.0.1.jar is an
> dependency of DSE4.8.3 Cassandra) as following exception.
>
> Do you know how to add external guava-16.0.1.jar in oozie class path?
>
> Thanks!
> -----
>
> >>> Invoking Spark class now >>>
>
>
> <<< Invocation of Main class completed <<<
>
> Failing Oozie Launcher, Main class
> [org.apache.oozie.action.hadoop.SparkMain], main() threw exception,
> com.google.common.reflect.TypeToken.isPrimitive()Z
> java.lang.NoSuchMethodError:
> com.google.common.reflect.TypeToken.isPrimitive()Z
> at com.datastax.driver.core.TypeCodec.<init>(TypeCodec.java:142)
> at com.datastax.driver.core.TypeCodec.<init>(TypeCodec.java:136)
> at com.datastax.driver.core.TypeCodec$BlobCodec.<init>(TypeCodec.java:609)
> at
> com.datastax.driver.core.TypeCodec$BlobCodec.<clinit>(TypeCodec.java:606)
> at com.datastax.driver.core.CodecRegistry.<clinit>(CodecRegistry.java:147)
> at
> com.datastax.driver.core.Configuration$Builder.build(Configuration.java:259)
> at
> com.datastax.driver.core.Cluster$Builder.getConfiguration(Cluster.java:1135)
> at com.datastax.driver.core.Cluster.<init>(Cluster.java:111)
> at com.datastax.driver.core.Cluster.buildFrom(Cluster.java:178)
> at com.datastax.driver.core.Cluster$Builder.build(Cluster.java:1152)
> at
> com.datastax.spark.connector.cql.DefaultConnectionFactory$.createCluster(CassandraConnectionFactory.scala:85)
> at
> com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:155)
> at
> com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:150)
> at
> com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:150)
> at
> com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:31)
> at
> com.datastax.spark.connector.cql.RefCountedCache.acquire(RefCountedCache.scala:56)
> at
> com.datastax.spark.connector.cql.CassandraConnector.openSession(CassandraConnector.scala:81)
> at
> com.datastax.spark.connector.cql.CassandraConnector.withSessionDo(CassandraConnector.scala:109)
> at
> com.datastax.spark.connector.cql.CassandraConnector.withClusterDo(CassandraConnector.scala:120)
> at com.datastax.spark.connector.cql.Schema$.fromCassandra(Schema.scala:241)
> at
> com.datastax.spark.connector.writer.TableWriter$.apply(TableWriter.scala:263)
> at
> com.datastax.spark.connector.RDDFunctions.saveToCassandra(RDDFunctions.scala:36)
> at TestCassandra$.main(TestCassandra.scala:44)
> at TestCassandra.main(TestCassandra.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> -----
>
> --
> Cheers,
> -----
> Big Data - Big Wisdom - Big Value
> --------------
> Michelle Zhang (张莉苹)
>
--
Cheers,
-----
Big Data - Big Wisdom - Big Value
--------------
Michelle Zhang (张莉苹)
