Re: Does Oozie support run sparkR with spark action?

[Peter Cseh]

Hi,

This exception is caused by a missing jar on the classpath.
The needed jars  should be added to the classpath in Oozie action. This
blogpost
describes
several ways to do it.

I've never tried to run a SparkR application from Oozie. I guess it can be
done, but in the current state it need some manual work:

According to Spark , the
SparkR libraries should be under  $SPARK_HOME/R/lib, and $R_HOME should be
also set for the job.
$SPARK_HOME is set to the current directory in Oozie after OOZIE-2482, and
you could add the SparkR stuff to Spark sharelib to make it available in
the action.
It's not guarantied that it will work after these steps, but there's a
chance. I would be delighted to hear about the result if you have the time
to try to make this work.

Thanks,
gp


On Tue, Nov 8, 2016 at 10:55 AM, Dongying Jiao 
wrote:

> Hi:
> I have an issue with oozie run sparkR, could you please help me?
> I try to run sparkR job through oozie in yarn-client mode. And I have
> installed R package in all my nodes.
>
> job.properties is like:
> nameNode=hdfs://XXX:8020
> jobTracker=XXX:8050
> master=yarn-client
> queueName=default
> oozie.use.system.libpath=true
> oozie.wf.application.path=/user/oozie/measurecountWF
>
> The workflow is like:
> 
> 
> 
> 
> oozie.launcher.yarn.app.mapreduce.am.env
> SPARK_HOME=
> 
> 
> 
> 
> 
> 
> ${jobTracker}
> ${nameNode}
> ${master}
> measurecountWF
> measurecount.R
>  --conf spark.driver.extraJavaOptions=
> 
>  
> 
>   
>   
>   
> Workflow failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]
> 
>   
>   
> 
>
> It failed with class not found exception.
>
> org.apache.spark.SparkException: Job aborted due to stage failure:
> Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3
> in stage 0.0 (TID 3, ): java.lang.ClassNotFoundException:
> com.cloudant.spark.common.JsonStoreRDDPartition
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.apache.spark.serializer.JavaDeserializationStream$$
> anon$1.resolveClass(JavaSerializer.scala:68)
> at java.io.ObjectInputStream.readNonProxyDesc(
> ObjectInputStream.java:1613)
> at java.io.ObjectInputStream.readClassDesc(
> ObjectInputStream.java:1518)
> at java.io.ObjectInputStream.readOrdinaryObject(
> ObjectInputStream.java:1774)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.
> java:1351)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInpu
> Calls: sql -> callJMethod -> invokeJava
> Execution halted
> Intercepting System.exit(1)
>
> Does oozie support run sparkR in spark action? Or we should only wrap
> it in ssh action?
>
> Thanks a lot
>



-- 
Peter Cseh
Software Engineer

Re: Problems with properties in Java action

[Peter Cseh]

Hey,

If you use the oozie.launcher prefix on a property, that property will be
applied to the Launcher MR job, that launches the action.
In your example if you set mapreduce.job.complete.cancel.delegation.tokens
to false in the action's configuration then Sqoop will get this property,
but the Launcher Mapper job isn't. If you set oozie.launcher.
mapreduce.job.complete.cancel.delegation.tokens to false there, then the
launcher job gets it.

I hope this helps,
gp

On Mon, Nov 7, 2016 at 11:38 AM, Андрей Ривкин  wrote:

> Maybe u could tell me some books or where in documentation I can read about
> it?
>
> Regards,
> Andrey
>
> 2016-11-02 10:02 GMT+03:00 Андрей Ривкин :
>
> > Hello everyone,
> >
> > Could somebody explain me how properties in oozie work?
> >
> > I've got java action on CDH 5.3 which generates a lot Sqoop actions and
> we
> > have problem with delegation tokens (https://issues.apache.org/jir
> > a/browse/YARN-2964). This job works for 20 mins and always fails with
> > delegation token not found in cache when tring to agregate logs. So we
> even
> > can't see logs.
> >
> > I've tried to set "mapreduce.job.complete.cancel.delegation.tokens" to
> > false in java action and in whole workflow, but it didn't set.
> > Then I've tried to set some custom propertie, for examle
> > "some.custom.property=true", but it also didn't set.
> > Then I've tried to change some oozie property, for example
> > "oozie.launcher.mapreduce.map.memory.mb" and it worked.
> >
> > So I'm confused how oozie is working with job properties.
> >
> > I'm checking all properties in JobHistory -> configuration.
> >
> >
> >
> > Regards,
> > Andrey
> >
>



-- 
Peter Cseh
Software Engineer