Re: Spark 2.3 in oozie

[purna pradeep]

I’m able to compile successfully with after adding this override option

-Dspark.scala.binary.version=2.11

Dspark.version = 2.3.0

But when I’m running a spark action with spark-pi example jar against
Kubernetes master I’m getting below error in stderr log


*Error:Could not load KUBERNETES classes.This copy of spark may not have
been compiled with Kubernetes support*

Below is my workflow.xml

<*spark xmlns="uri:oozie:spark-action:1.0">*

*${resourceManager}*

*${nameNode}*

*k8s://<***.com>*

*Python-Spark-Pi*

*spark-examples_2.11-2.3.0.jar*

*--class org.apache.spark.examples.SparkPi --conf
spark.executor.instances=2 --conf spark.kubernetes.namespace=spark --conf
spark.kubernetes.authenticate.driver.serviceAccountName=spark --conf
spark.kubernetes.container.image=artifactory.cloud.capitalone.com/kubespark/spark-quantum:v2.3.0

--conf spark.kubernetes.node.selector.node-role.kubernetes.io/worker=true

--conf
spark.kubernetes.driver.label.application=is1-driver --conf
spark.kubernetes.executor.label.application=is1-exec*utor
local:///opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar




Is this because of uri:oozie:spark-action:1.0 in spark xml tag? Does it
needs to be spark-action:2.0 as I’m using spark 2.3?


Please suggest!


On Tue, May 15, 2018 at 12:43 PM Peter Cseh  wrote:

> I think the error is related to the Scala version being present in the
> artifact name.
> I'll take a look at this tomorrow.
> Gp
>
> On Tue, May 15, 2018, 18:28 Artem Ervits  wrote:
>
> > Did you run
> > mvn clean install first on the parent directory?
> >
> > On Tue, May 15, 2018, 11:35 AM purna pradeep 
> > wrote:
> >
> > > Thanks peter,
> > >
> > > I have tried changing Dspark.version to 2.3.0 and compiled oozie I’m
> > > getting below error from oozie examples
> > >
> > >
> > > *ERROR] Failed to execute goal on project oozie-examples: Could not
> > resolve
> > > dependencies for project org.apache.oozie:oozie-examples:jar:5.0.0:
> Could
> > > not find artifact org.apache.spark:spark-core_2.10:jar:2.3.0 in
> > resolution
> > > *
> > >
> > > On Tue, May 15, 2018 at 11:14 AM Peter Cseh 
> > wrote:
> > >
> > > > Oozie has a spark-2 profile that is currently hard-coded to Spark
> 2.1:
> > > > https://github.com/apache/oozie/blob/master/pom.xml#L1983
> > > > I'm sure if you overwrite the -Dspark.version and compile Oozie that
> > way
> > > it
> > > > will work.
> > > > gp
> > > >
> > > >
> > > > On Tue, May 15, 2018 at 5:07 PM, purna pradeep <
> > purna2prad...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > Does oozie supports spark 2.3? Or will it even care of the spark
> > > version
> > > > >
> > > > > I want to use spark action
> > > > >
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Purna
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *Peter Cseh *| Software Engineer
> > > > cloudera.com 
> > > >
> > > > [image: Cloudera] 
> > > >
> > > > [image: Cloudera on Twitter]  [image:
> > > > Cloudera on Facebook]  [image:
> > > Cloudera
> > > > on LinkedIn] 
> > > > --
> > > >
> > >
> >
>

Re: Spark 2.3 in oozie

[Peter Cseh]

I think the error is related to the Scala version being present in the
artifact name.
I'll take a look at this tomorrow.
Gp

On Tue, May 15, 2018, 18:28 Artem Ervits  wrote:

> Did you run
> mvn clean install first on the parent directory?
>
> On Tue, May 15, 2018, 11:35 AM purna pradeep 
> wrote:
>
> > Thanks peter,
> >
> > I have tried changing Dspark.version to 2.3.0 and compiled oozie I’m
> > getting below error from oozie examples
> >
> >
> > *ERROR] Failed to execute goal on project oozie-examples: Could not
> resolve
> > dependencies for project org.apache.oozie:oozie-examples:jar:5.0.0: Could
> > not find artifact org.apache.spark:spark-core_2.10:jar:2.3.0 in
> resolution
> > *
> >
> > On Tue, May 15, 2018 at 11:14 AM Peter Cseh 
> wrote:
> >
> > > Oozie has a spark-2 profile that is currently hard-coded to Spark 2.1:
> > > https://github.com/apache/oozie/blob/master/pom.xml#L1983
> > > I'm sure if you overwrite the -Dspark.version and compile Oozie that
> way
> > it
> > > will work.
> > > gp
> > >
> > >
> > > On Tue, May 15, 2018 at 5:07 PM, purna pradeep <
> purna2prad...@gmail.com>
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > Does oozie supports spark 2.3? Or will it even care of the spark
> > version
> > > >
> > > > I want to use spark action
> > > >
> > > >
> > > >
> > > > Thanks,
> > > > Purna
> > > >
> > >
> > >
> > >
> > > --
> > > *Peter Cseh *| Software Engineer
> > > cloudera.com 
> > >
> > > [image: Cloudera] 
> > >
> > > [image: Cloudera on Twitter]  [image:
> > > Cloudera on Facebook]  [image:
> > Cloudera
> > > on LinkedIn] 
> > > --
> > >
> >
>

Re: Spark 2.3 in oozie

[Artem Ervits]

Did you run
mvn clean install first on the parent directory?

On Tue, May 15, 2018, 11:35 AM purna pradeep 
wrote:

> Thanks peter,
>
> I have tried changing Dspark.version to 2.3.0 and compiled oozie I’m
> getting below error from oozie examples
>
>
> *ERROR] Failed to execute goal on project oozie-examples: Could not resolve
> dependencies for project org.apache.oozie:oozie-examples:jar:5.0.0: Could
> not find artifact org.apache.spark:spark-core_2.10:jar:2.3.0 in resolution
> *
>
> On Tue, May 15, 2018 at 11:14 AM Peter Cseh  wrote:
>
> > Oozie has a spark-2 profile that is currently hard-coded to Spark 2.1:
> > https://github.com/apache/oozie/blob/master/pom.xml#L1983
> > I'm sure if you overwrite the -Dspark.version and compile Oozie that way
> it
> > will work.
> > gp
> >
> >
> > On Tue, May 15, 2018 at 5:07 PM, purna pradeep 
> > wrote:
> >
> > > Hello,
> > >
> > > Does oozie supports spark 2.3? Or will it even care of the spark
> version
> > >
> > > I want to use spark action
> > >
> > >
> > >
> > > Thanks,
> > > Purna
> > >
> >
> >
> >
> > --
> > *Peter Cseh *| Software Engineer
> > cloudera.com 
> >
> > [image: Cloudera] 
> >
> > [image: Cloudera on Twitter]  [image:
> > Cloudera on Facebook]  [image:
> Cloudera
> > on LinkedIn] 
> > --
> >
>

Re: Spark 2.3 in oozie

[purna pradeep]

Thanks peter,

I have tried changing Dspark.version to 2.3.0 and compiled oozie I’m
getting below error from oozie examples


*ERROR] Failed to execute goal on project oozie-examples: Could not resolve
dependencies for project org.apache.oozie:oozie-examples:jar:5.0.0: Could
not find artifact org.apache.spark:spark-core_2.10:jar:2.3.0 in resolution *

On Tue, May 15, 2018 at 11:14 AM Peter Cseh  wrote:

> Oozie has a spark-2 profile that is currently hard-coded to Spark 2.1:
> https://github.com/apache/oozie/blob/master/pom.xml#L1983
> I'm sure if you overwrite the -Dspark.version and compile Oozie that way it
> will work.
> gp
>
>
> On Tue, May 15, 2018 at 5:07 PM, purna pradeep 
> wrote:
>
> > Hello,
> >
> > Does oozie supports spark 2.3? Or will it even care of the spark version
> >
> > I want to use spark action
> >
> >
> >
> > Thanks,
> > Purna
> >
>
>
>
> --
> *Peter Cseh *| Software Engineer
> cloudera.com 
>
> [image: Cloudera] 
>
> [image: Cloudera on Twitter]  [image:
> Cloudera on Facebook]  [image: Cloudera
> on LinkedIn] 
> --
>

Re: Spark 2.3 in oozie

[Peter Cseh]

Oozie has a spark-2 profile that is currently hard-coded to Spark 2.1:
https://github.com/apache/oozie/blob/master/pom.xml#L1983
I'm sure if you overwrite the -Dspark.version and compile Oozie that way it
will work.
gp


On Tue, May 15, 2018 at 5:07 PM, purna pradeep 
wrote:

> Hello,
>
> Does oozie supports spark 2.3? Or will it even care of the spark version
>
> I want to use spark action
>
>
>
> Thanks,
> Purna
>



-- 
*Peter Cseh *| Software Engineer
cloudera.com 

[image: Cloudera] 

[image: Cloudera on Twitter]  [image:
Cloudera on Facebook]  [image: Cloudera
on LinkedIn] 
--

Spark 2.3 in oozie

[purna pradeep]

Hello,

Does oozie supports spark 2.3? Or will it even care of the spark version

I want to use spark action



Thanks,
Purna

Re: Oozie for spark jobs without Hadoop

[purna pradeep]

Thanks Andras,

Also I also would like to know if oozie supports Aws S3 as input events to
poll for a dependency file before kicking off a spark action


For example: I don’t want to kick off a spark action until a file is
arrived on a given AWS s3 location

On Tue, May 15, 2018 at 10:17 AM Andras Piros 
wrote:

> Hi,
>
> Oozie needs HDFS to store workflow, coordinator, or bundle definitions, as
> well as sharelib files in a safe, distributed and scalable way. Oozie needs
> YARN to run almost all of its actions, Spark action being no exception.
>
> At the moment it's not feasible to install Oozie without those Hadoop
> components. How to install Oozie please *find here
> *.
>
> Regards,
>
> Andras
>
> On Tue, May 15, 2018 at 4:11 PM, purna pradeep 
> wrote:
>
> > Hi,
> >
> > Would like to know if I can use sparkaction in oozie without having
> Hadoop
> > cluster?
> >
> > I want to use oozie to schedule spark jobs on Kubernetes cluster
> >
> > I’m a beginner in oozie
> >
> > Thanks
> >
>

Re: Oozie for spark jobs without Hadoop

[Andras Piros]

Hi,

Oozie needs HDFS to store workflow, coordinator, or bundle definitions, as
well as sharelib files in a safe, distributed and scalable way. Oozie needs
YARN to run almost all of its actions, Spark action being no exception.

At the moment it's not feasible to install Oozie without those Hadoop
components. How to install Oozie please *find here
*.

Regards,

Andras

On Tue, May 15, 2018 at 4:11 PM, purna pradeep 
wrote:

> Hi,
>
> Would like to know if I can use sparkaction in oozie without having Hadoop
> cluster?
>
> I want to use oozie to schedule spark jobs on Kubernetes cluster
>
> I’m a beginner in oozie
>
> Thanks
>