Thanks @Neils and @Kyle.
@Neils - I agree, I don't want to copy Hadoop configurations to my airflow VM.
In this (Using SSHOperator) case, Airflow would just be receiving std output
right as opposed to driver logs?
@Kyle - If you can, then it would definitely be useful to have LivyOperators to
Airflow.
Regards,
Kaxil
On 22/06/2018, 13:34, "Niels Zeilemaker" <ni...@zeilemaker.nl> wrote:
Hi Kaxil,
I would recommend using the SSHOperator to start the Spark Job on the
master node of the HDInsight cluster.
This avoids the problems associated with Livy, and doesn't require you to
open ports/copy the hadoop configuration to your airflow machine.
Niels
2018-06-22 14:17 GMT+02:00 Naik Kaxil <k.n...@reply.com>:
> Hi all,
>
>
>
> Has anyone used the SparkSubmitOperator to submit Spark jobs on Azure
> HDInsight cluster? Are you using Livy or spark-submit to run remote Spark
> jobs?
>
>
>
> Regards,
>
> Kaxil
>
>
> Kaxil Naik
>
> Data Reply
> 2nd Floor, Nova South
> 160 Victoria Street, Westminster
>
<https://maps.google.com/?q=160+Victoria+Street,+Westminster+%0D%0ALondon+SW1E+5LB+-+UK&entry=gmail&source=g>
> London SW1E 5LB - UK
> phone: +44 (0)20 7730 6000
> k.n...@reply.com
> www.reply.com
>
> [image: Data Reply]
>
Kaxil Naik
Data Reply
2nd Floor, Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK
phone: +44 (0)20 7730 6000
k.n...@reply.com
www.reply.com
