Thanks Robert. I tried your methods, and replied you with issues in following. http://community.cloudera.com/t5/Batch-Processing-and-Workflow/how-to-add-external-guava-16-0-1-jar-in-CDH-oozie-classpath/m-p/37803#U37803
On Wed, Feb 24, 2016 at 9:10 PM, Robert Kanter <[email protected]> wrote: > I replied on the Cloudera forum thread with more details and things to > check. Let's keep the conversation all in one place so it's easier to > follow. > > > http://community.cloudera.com/t5/Batch-Processing-and-Workflow/how-to-add-external-guava-16-0-1-jar-in-CDH-oozie-classpath/m-p/37803#U37803 > > On Wed, Feb 24, 2016 at 3:40 PM, Liping Zhang <[email protected]> > wrote: > >> Is there any example about how to add external jar into oozie when >> running spark action in oozie? >> >> On Wed, Feb 24, 2016 at 3:37 PM, Liping Zhang <[email protected]> >> wrote: >> >>> Actually I'm still a little confued about the 4 ways mentioned in "One >>> Last Thing" in >>> http://blog.cloudera.com/blog/2014/05/how-to-use-the-sharelib-in-apache-oozie-cdh-5/ >>> I tried all the ways, but all didn't work.(I'm using CDH Hue and oozie >>> workflow) Following were what I tried with the 4 ways: >>> >>> For way 1: >>> It recommended "oozie.libpath=/path/to/jars,another/path/to/jars" >>> I add >>> oozie.libpath=hdfs://ip-10-0-4-248.us-west-1.compute.internal:8020/user/oozie/share/lib/lib_20151201085935/spark >>> or >>> oozie.libpath=hdfs://ip-10-0-4-248.us-west-1.compute.internal:8020/user/oozie/share/lib/lib_20151201085935/spark/guava-16.0.1.jar >>> and oozie.use.system.libpath=true is by default. >>> Both don't work. >>> >>> For way 2: >>> I added guava-16.0.1.jar into “lib” next to current workspace >>> workflow.xml in HDFS, it doesn't work. >>> >>> For way 3: >>> I can not find any <archive> tag in a Spark action with the path to a >>> single jar, so I have no way to try way3. >>> >>> For way 4: >>> I added guava-16.0.1.jar to the ShareLib (e.g. >>> hdfs://ip-10-0-4-248.us-west-1.compute.internal:8020/user/oozie/share/lib/lib_20151201085935/spark) >>> and set oozie.use.system.libpath=true in job.properties, it still doesn't >>> work. >>> >>> Could you please give any suggestion? Thanks very much for any of your >>> help! I appreciated! >>> >>> >>> On Wed, Feb 24, 2016 at 3:13 PM, Liping Zhang <[email protected]> >>> wrote: >>> >>>> Thanks Robert! >>>> >>>> I tried the way 1, 2, 4 introduced in >>>> http://blog.cloudera.com/blog/2014/05/how-to-use-the-sharelib-in-apache-oozie-cdh-5/, >>>> but still doesn't work. >>>> >>>> >>>> >>>> Here is the detailed I posted in CDH support forum, but no one answers >>>> so far. >>>> >>>> >>>> http://community.cloudera.com/t5/Batch-Processing-and-Workflow/how-to-add-external-guava-16-0-1-jar-in-CDH-oozie-classpath/m-p/37803#U37803 >>>> >>>> >>>> Can you give an detailed example with commands steps? Thanks! >>>> >>>> On Wed, Feb 24, 2016 at 2:21 PM, Liping Zhang <[email protected]> >>>> wrote: >>>> >>>>> Thanks very much Robert for you quick answer! >>>>> >>>>> I upload the guava-16.0.1.jar into HDFS >>>>> /user/oozie/share/lib/lib_20151201085935/spark >>>>> dir, and restart oozie from CM, under the dir, there is only one >>>>> guava-16.0.1.jar not guava-14.0.1.jar in HDFS /user/oozie/share/lib/ >>>>> lib_20151201085935/spark. >>>>> >>>>> However, it still has the same "main() threw exception, >>>>> com.google.common.reflect.TypeToken.isPrimitive()Z" exception. Is >>>>> there anything else I need to configure in workflow.xml or in >>>>> job.properties in HDFS workspace? >>>>> >>>>> Appreciated if any help provided! >>>>> >>>>> On Wed, Feb 24, 2016 at 2:12 PM, Robert Kanter <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Liping, >>>>>> >>>>>> If you change jars, or anything with the sharelib directories, you >>>>>> have to >>>>>> either restart the Oozie server or run the 'oozie admin >>>>>> -sharelibupdate' >>>>>> command for it to notice. >>>>>> >>>>>> That said, CDH Spark requires Guava 14.0.1, which is what's in the >>>>>> Spark >>>>>> sharelib directory already. If you simply put Guava 16.0.1 in there, >>>>>> you >>>>>> may run into other problems. >>>>>> >>>>>> >>>>>> - Robert >>>>>> >>>>>> On Wed, Feb 24, 2016 at 12:34 PM, Liping Zhang <[email protected] >>>>>> > >>>>>> wrote: >>>>>> >>>>>> > I added guava-16.0.1.jar into >>>>>> > HDFS /user/oozie/share/lib/lib_20151201085935/spark dir, and chown >>>>>> as >>>>>> > "oozie:oozie", chmod 777, but it still could not find the jar. >>>>>> > >>>>>> > job.properties: >>>>>> > >>>>>> > oozie.use.system.libpath=True >>>>>> > security_enabled=False >>>>>> > dryrun=False >>>>>> > jobTracker=ip-10-0-4-248.us-west-1.compute.internal:8032 >>>>>> > nameNode=hdfs://ip-10-0-4-248.us-west-1.compute.internal:8020 >>>>>> > >>>>>> > >>>>>> > workflow.xml: >>>>>> > <workflow-app name="sparktest-cassandra" >>>>>> xmlns="uri:oozie:workflow:0.5"> >>>>>> > <start to="spark-b23b"/> >>>>>> > <kill name="Kill"> >>>>>> > <message>Action failed, error >>>>>> > message[${wf:errorMessage(wf:lastErrorNode())}]</message> >>>>>> > </kill> >>>>>> > <action name="spark-b23b"> >>>>>> > <spark xmlns="uri:oozie:spark-action:0.1"> >>>>>> > <job-tracker>${jobTracker}</job-tracker> >>>>>> > <name-node>${nameNode}</name-node> >>>>>> > <master>local[4]</master> >>>>>> > <mode>client</mode> >>>>>> > <name>sparktest-cassandra</name> >>>>>> > <class>TestCassandra</class> >>>>>> > <jar>lib/sparktest.jar</jar> >>>>>> > <spark-opts>--driver-class-path >>>>>> > /opt/cloudera/parcels/CDH/jars/guava-16.0.1.jar --jars >>>>>> > lib/*.jar</spark-opts> >>>>>> > <arg>s3n://gridx-output/sparktest/ </arg> >>>>>> > <arg>10</arg> >>>>>> > <arg>3</arg> >>>>>> > <arg>2</arg> >>>>>> > </spark> >>>>>> > <ok to="End"/> >>>>>> > <error to="Kill"/> >>>>>> > </action> >>>>>> > <end name="End"/> >>>>>> > </workflow-app> >>>>>> > >>>>>> > Appreciated if any help! >>>>>> > >>>>>> > On Wed, Feb 24, 2016 at 12:34 PM, Liping Zhang < >>>>>> [email protected]> >>>>>> > wrote: >>>>>> > >>>>>> > > Dear, >>>>>> > > >>>>>> > > We used CDH5.5.0 Hue Oozie to run Spark action, and in the spark >>>>>> job >>>>>> > > action, the job used spark-cassandra-connector_2.10-1.5.0-M2.jar. >>>>>> > > >>>>>> > > The job can run successfully with spark-submit command, but it >>>>>> was failed >>>>>> > > to run in oozie. >>>>>> > > >>>>>> > > It said oozie can not find guava-16.0.1.jar(guava-16.0.1.jar is an >>>>>> > > dependency of DSE4.8.3 Cassandra) as following exception. >>>>>> > > >>>>>> > > Do you know how to add external guava-16.0.1.jar in oozie class >>>>>> path? >>>>>> > > >>>>>> > > Thanks! >>>>>> > > ----- >>>>>> > > >>>>>> > > >>> Invoking Spark class now >>> >>>>>> > > >>>>>> > > >>>>>> > > <<< Invocation of Main class completed <<< >>>>>> > > >>>>>> > > Failing Oozie Launcher, Main class >>>>>> > > [org.apache.oozie.action.hadoop.SparkMain], main() threw >>>>>> exception, >>>>>> > > com.google.common.reflect.TypeToken.isPrimitive()Z >>>>>> > > java.lang.NoSuchMethodError: >>>>>> > > com.google.common.reflect.TypeToken.isPrimitive()Z >>>>>> > > at com.datastax.driver.core.TypeCodec.<init>(TypeCodec.java:142) >>>>>> > > at com.datastax.driver.core.TypeCodec.<init>(TypeCodec.java:136) >>>>>> > > at >>>>>> > >>>>>> com.datastax.driver.core.TypeCodec$BlobCodec.<init>(TypeCodec.java:609) >>>>>> > > at >>>>>> > > >>>>>> com.datastax.driver.core.TypeCodec$BlobCodec.<clinit>(TypeCodec.java:606) >>>>>> > > at >>>>>> > >>>>>> com.datastax.driver.core.CodecRegistry.<clinit>(CodecRegistry.java:147) >>>>>> > > at >>>>>> > > >>>>>> > >>>>>> com.datastax.driver.core.Configuration$Builder.build(Configuration.java:259) >>>>>> > > at >>>>>> > > >>>>>> > >>>>>> com.datastax.driver.core.Cluster$Builder.getConfiguration(Cluster.java:1135) >>>>>> > > at com.datastax.driver.core.Cluster.<init>(Cluster.java:111) >>>>>> > > at com.datastax.driver.core.Cluster.buildFrom(Cluster.java:178) >>>>>> > > at >>>>>> com.datastax.driver.core.Cluster$Builder.build(Cluster.java:1152) >>>>>> > > at >>>>>> > > >>>>>> > >>>>>> com.datastax.spark.connector.cql.DefaultConnectionFactory$.createCluster(CassandraConnectionFactory.scala:85) >>>>>> > > at >>>>>> > > >>>>>> > >>>>>> com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:155) >>>>>> > > at >>>>>> > > >>>>>> > >>>>>> com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:150) >>>>>> > > at >>>>>> > > >>>>>> > >>>>>> com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:150) >>>>>> > > at >>>>>> > > >>>>>> > >>>>>> com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:31) >>>>>> > > at >>>>>> > > >>>>>> > >>>>>> com.datastax.spark.connector.cql.RefCountedCache.acquire(RefCountedCache.scala:56) >>>>>> > > at >>>>>> > > >>>>>> > >>>>>> com.datastax.spark.connector.cql.CassandraConnector.openSession(CassandraConnector.scala:81) >>>>>> > > at >>>>>> > > >>>>>> > >>>>>> com.datastax.spark.connector.cql.CassandraConnector.withSessionDo(CassandraConnector.scala:109) >>>>>> > > at >>>>>> > > >>>>>> > >>>>>> com.datastax.spark.connector.cql.CassandraConnector.withClusterDo(CassandraConnector.scala:120) >>>>>> > > at >>>>>> > >>>>>> com.datastax.spark.connector.cql.Schema$.fromCassandra(Schema.scala:241) >>>>>> > > at >>>>>> > > >>>>>> > >>>>>> com.datastax.spark.connector.writer.TableWriter$.apply(TableWriter.scala:263) >>>>>> > > at >>>>>> > > >>>>>> > >>>>>> com.datastax.spark.connector.RDDFunctions.saveToCassandra(RDDFunctions.scala:36) >>>>>> > > at TestCassandra$.main(TestCassandra.scala:44) >>>>>> > > at TestCassandra.main(TestCassandra.scala) >>>>>> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>> > > >>>>>> > > ----- >>>>>> > > >>>>>> > > -- >>>>>> > > Cheers, >>>>>> > > ----- >>>>>> > > Big Data - Big Wisdom - Big Value >>>>>> > > -------------- >>>>>> > > Michelle Zhang (张莉苹) >>>>>> > > >>>>>> > >>>>>> > >>>>>> > >>>>>> > -- >>>>>> > Cheers, >>>>>> > ----- >>>>>> > Big Data - Big Wisdom - Big Value >>>>>> > -------------- >>>>>> > Michelle Zhang (张莉苹) >>>>>> > >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Cheers, >>>>> ----- >>>>> Big Data - Big Wisdom - Big Value >>>>> -------------- >>>>> Michelle Zhang (张莉苹) >>>>> >>>> >>>> >>>> >>>> -- >>>> Cheers, >>>> ----- >>>> Big Data - Big Wisdom - Big Value >>>> -------------- >>>> Michelle Zhang (张莉苹) >>>> >>> >>> >>> >>> -- >>> Cheers, >>> ----- >>> Big Data - Big Wisdom - Big Value >>> -------------- >>> Michelle Zhang (张莉苹) >>> >> >> >> >> -- >> Cheers, >> ----- >> Big Data - Big Wisdom - Big Value >> -------------- >> Michelle Zhang (张莉苹) >> > > -- Cheers, ----- Big Data - Big Wisdom - Big Value -------------- Michelle Zhang (张莉苹)
