Re: SparkSubmitOperator with Azure HDInsight

[Niels Zeilemaker]

If you run in non-cluster mode, the driver logs would be written to stdout.
So airflow would capture those.


Op vr 22 jun. 2018 18:04 schreef Naik Kaxil :

> Thanks @Neils and @Kyle.
>
> @Neils - I agree, I don't want to copy Hadoop configurations to my airflow
> VM. In this (Using SSHOperator) case, Airflow would just be receiving std
> output right as opposed to driver logs?
>
> @Kyle - If you can, then it would definitely be useful to have
> LivyOperators to Airflow.
>
> Regards,
> Kaxil
>
> On 22/06/2018, 13:34, "Niels Zeilemaker"  wrote:
>
> Hi Kaxil,
>
> I would recommend using the SSHOperator to start the Spark Job on the
> master node of the HDInsight cluster.
> This avoids the problems associated with Livy, and doesn't require you
> to
> open ports/copy the hadoop configuration to your airflow machine.
>
> Niels
>
> 2018-06-22 14:17 GMT+02:00 Naik Kaxil :
>
> > Hi all,
> >
> >
> >
> > Has anyone used the SparkSubmitOperator to submit Spark jobs on Azure
> > HDInsight cluster? Are you using Livy or spark-submit to run remote
> Spark
> > jobs?
> >
> >
> >
> > Regards,
> >
> > Kaxil
> >
> >
> > Kaxil Naik
> >
> > Data Reply
> > 2nd Floor, Nova South
> > 160 Victoria Street, Westminster
> > <
> https://maps.google.com/?q=160+Victoria+Street,+Westminster+%0D%0ALondon+SW1E+5LB+-+UK=gmail=g
> >
> > London SW1E 5LB - UK
> > phone: +44 (0)20 7730 6000
> > k.n...@reply.com
> > www.reply.com
> >
> > [image: Data Reply]
> >
>
>
>
>
>
>
> Kaxil Naik
>
> Data Reply
> 2nd Floor, Nova South
> 160 Victoria Street, Westminster
> London SW1E 5LB - UK
> phone: +44 (0)20 7730 6000
> k.n...@reply.com
> www.reply.com
>

Re: SparkSubmitOperator with Azure HDInsight

[Naik Kaxil]

Thanks @Neils and @Kyle.

@Neils - I agree, I don't want to copy Hadoop configurations to my airflow VM. 
In this (Using SSHOperator) case, Airflow would just be receiving std output 
right as opposed to driver logs?

@Kyle - If you can, then it would definitely be useful to have LivyOperators to 
Airflow.

Regards,
Kaxil 

On 22/06/2018, 13:34, "Niels Zeilemaker"  wrote:

Hi Kaxil,

I would recommend using the SSHOperator to start the Spark Job on the
master node of the HDInsight cluster.
This avoids the problems associated with Livy, and doesn't require you to
open ports/copy the hadoop configuration to your airflow machine.

Niels

2018-06-22 14:17 GMT+02:00 Naik Kaxil :

> Hi all,
>
>
>
> Has anyone used the SparkSubmitOperator to submit Spark jobs on Azure
> HDInsight cluster? Are you using Livy or spark-submit to run remote Spark
> jobs?
>
>
>
> Regards,
>
> Kaxil
>
>
> Kaxil Naik
>
> Data Reply
> 2nd Floor, Nova South
> 160 Victoria Street, Westminster
> 

> London SW1E 5LB - UK
> phone: +44 (0)20 7730 6000
> k.n...@reply.com
> www.reply.com
>
> [image: Data Reply]
>






Kaxil Naik 

Data Reply
2nd Floor, Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK 
phone: +44 (0)20 7730 6000
k.n...@reply.com
www.reply.com

Re: SparkSubmitOperator with Azure HDInsight

[Niels Zeilemaker]

Hi Kaxil,

I would recommend using the SSHOperator to start the Spark Job on the
master node of the HDInsight cluster.
This avoids the problems associated with Livy, and doesn't require you to
open ports/copy the hadoop configuration to your airflow machine.

Niels

2018-06-22 14:17 GMT+02:00 Naik Kaxil :

> Hi all,
>
>
>
> Has anyone used the SparkSubmitOperator to submit Spark jobs on Azure
> HDInsight cluster? Are you using Livy or spark-submit to run remote Spark
> jobs?
>
>
>
> Regards,
>
> Kaxil
>
>
> Kaxil Naik
>
> Data Reply
> 2nd Floor, Nova South
> 160 Victoria Street, Westminster
> 
> London SW1E 5LB - UK
> phone: +44 (0)20 7730 6000
> k.n...@reply.com
> www.reply.com
>
> [image: Data Reply]
>

Re: SparkSubmitOperator with Azure HDInsight

[Kyle Hamlin]

I haven’t used any Azure products but I did build a Livy hook and operator
so I could submit concurrent spark jobs to EMR clusters. I was planning on
contributing the code, but it’s kinda a pain haha. If your interested I can
take another stab at getting the Livy hook and operator contributed.

On Fri, Jun 22, 2018 at 8:18 AM Naik Kaxil  wrote:

> Hi all,
>
>
>
> Has anyone used the SparkSubmitOperator to submit Spark jobs on Azure
> HDInsight cluster? Are you using Livy or spark-submit to run remote Spark
> jobs?
>
>
>
> Regards,
>
> Kaxil
>
>
> Kaxil Naik
>
> Data Reply
> 2nd Floor, Nova South
> 160 Victoria Street, Westminster
> 
> London SW1E 5LB - UK
> 
> phone: +44 (0)20 7730 6000
> k.n...@reply.com
> www.reply.com
>
> [image: Data Reply]
>
-- 
Kyle Hamlin

SparkSubmitOperator with Azure HDInsight

[Naik Kaxil]

Hi all,

Has anyone used the SparkSubmitOperator to submit Spark jobs on Azure HDInsight 
cluster? Are you using Livy or spark-submit to run remote Spark jobs?

Regards,
Kaxil


Kaxil Naik

Data Reply
2nd Floor, Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK
phone: +44 (0)20 7730 6000
k.n...@reply.com
www.reply.com

[Data Reply]