Thanks very much Robert for you quick answer! I upload the guava-16.0.1.jar into HDFS /user/oozie/share/lib/lib_20151201085935/spark dir, and restart oozie from CM, under the dir, there is only one guava-16.0.1.jar not guava-14.0.1.jar in HDFS /user/oozie/share/lib/ lib_20151201085935/spark.
However, it still has the same "main() threw exception, com.google.common.reflect.TypeToken.isPrimitive()Z" exception. Is there anything else I need to configure in workflow.xml or in job.properties in HDFS workspace? Appreciated if any help provided! On Wed, Feb 24, 2016 at 2:12 PM, Robert Kanter <[email protected]> wrote: > Hi Liping, > > If you change jars, or anything with the sharelib directories, you have to > either restart the Oozie server or run the 'oozie admin -sharelibupdate' > command for it to notice. > > That said, CDH Spark requires Guava 14.0.1, which is what's in the Spark > sharelib directory already. If you simply put Guava 16.0.1 in there, you > may run into other problems. > > > - Robert > > On Wed, Feb 24, 2016 at 12:34 PM, Liping Zhang <[email protected]> > wrote: > > > I added guava-16.0.1.jar into > > HDFS /user/oozie/share/lib/lib_20151201085935/spark dir, and chown as > > "oozie:oozie", chmod 777, but it still could not find the jar. > > > > job.properties: > > > > oozie.use.system.libpath=True > > security_enabled=False > > dryrun=False > > jobTracker=ip-10-0-4-248.us-west-1.compute.internal:8032 > > nameNode=hdfs://ip-10-0-4-248.us-west-1.compute.internal:8020 > > > > > > workflow.xml: > > <workflow-app name="sparktest-cassandra" xmlns="uri:oozie:workflow:0.5"> > > <start to="spark-b23b"/> > > <kill name="Kill"> > > <message>Action failed, error > > message[${wf:errorMessage(wf:lastErrorNode())}]</message> > > </kill> > > <action name="spark-b23b"> > > <spark xmlns="uri:oozie:spark-action:0.1"> > > <job-tracker>${jobTracker}</job-tracker> > > <name-node>${nameNode}</name-node> > > <master>local[4]</master> > > <mode>client</mode> > > <name>sparktest-cassandra</name> > > <class>TestCassandra</class> > > <jar>lib/sparktest.jar</jar> > > <spark-opts>--driver-class-path > > /opt/cloudera/parcels/CDH/jars/guava-16.0.1.jar --jars > > lib/*.jar</spark-opts> > > <arg>s3n://gridx-output/sparktest/ </arg> > > <arg>10</arg> > > <arg>3</arg> > > <arg>2</arg> > > </spark> > > <ok to="End"/> > > <error to="Kill"/> > > </action> > > <end name="End"/> > > </workflow-app> > > > > Appreciated if any help! > > > > On Wed, Feb 24, 2016 at 12:34 PM, Liping Zhang <[email protected]> > > wrote: > > > > > Dear, > > > > > > We used CDH5.5.0 Hue Oozie to run Spark action, and in the spark job > > > action, the job used spark-cassandra-connector_2.10-1.5.0-M2.jar. > > > > > > The job can run successfully with spark-submit command, but it was > failed > > > to run in oozie. > > > > > > It said oozie can not find guava-16.0.1.jar(guava-16.0.1.jar is an > > > dependency of DSE4.8.3 Cassandra) as following exception. > > > > > > Do you know how to add external guava-16.0.1.jar in oozie class path? > > > > > > Thanks! > > > ----- > > > > > > >>> Invoking Spark class now >>> > > > > > > > > > <<< Invocation of Main class completed <<< > > > > > > Failing Oozie Launcher, Main class > > > [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, > > > com.google.common.reflect.TypeToken.isPrimitive()Z > > > java.lang.NoSuchMethodError: > > > com.google.common.reflect.TypeToken.isPrimitive()Z > > > at com.datastax.driver.core.TypeCodec.<init>(TypeCodec.java:142) > > > at com.datastax.driver.core.TypeCodec.<init>(TypeCodec.java:136) > > > at > > com.datastax.driver.core.TypeCodec$BlobCodec.<init>(TypeCodec.java:609) > > > at > > > > com.datastax.driver.core.TypeCodec$BlobCodec.<clinit>(TypeCodec.java:606) > > > at > > com.datastax.driver.core.CodecRegistry.<clinit>(CodecRegistry.java:147) > > > at > > > > > > com.datastax.driver.core.Configuration$Builder.build(Configuration.java:259) > > > at > > > > > > com.datastax.driver.core.Cluster$Builder.getConfiguration(Cluster.java:1135) > > > at com.datastax.driver.core.Cluster.<init>(Cluster.java:111) > > > at com.datastax.driver.core.Cluster.buildFrom(Cluster.java:178) > > > at com.datastax.driver.core.Cluster$Builder.build(Cluster.java:1152) > > > at > > > > > > com.datastax.spark.connector.cql.DefaultConnectionFactory$.createCluster(CassandraConnectionFactory.scala:85) > > > at > > > > > > com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:155) > > > at > > > > > > com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:150) > > > at > > > > > > com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:150) > > > at > > > > > > com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:31) > > > at > > > > > > com.datastax.spark.connector.cql.RefCountedCache.acquire(RefCountedCache.scala:56) > > > at > > > > > > com.datastax.spark.connector.cql.CassandraConnector.openSession(CassandraConnector.scala:81) > > > at > > > > > > com.datastax.spark.connector.cql.CassandraConnector.withSessionDo(CassandraConnector.scala:109) > > > at > > > > > > com.datastax.spark.connector.cql.CassandraConnector.withClusterDo(CassandraConnector.scala:120) > > > at > > com.datastax.spark.connector.cql.Schema$.fromCassandra(Schema.scala:241) > > > at > > > > > > com.datastax.spark.connector.writer.TableWriter$.apply(TableWriter.scala:263) > > > at > > > > > > com.datastax.spark.connector.RDDFunctions.saveToCassandra(RDDFunctions.scala:36) > > > at TestCassandra$.main(TestCassandra.scala:44) > > > at TestCassandra.main(TestCassandra.scala) > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > > > ----- > > > > > > -- > > > Cheers, > > > ----- > > > Big Data - Big Wisdom - Big Value > > > -------------- > > > Michelle Zhang (张莉苹) > > > > > > > > > > > -- > > Cheers, > > ----- > > Big Data - Big Wisdom - Big Value > > -------------- > > Michelle Zhang (张莉苹) > > > -- Cheers, ----- Big Data - Big Wisdom - Big Value -------------- Michelle Zhang (张莉苹)
