Hi Donald, As per my understanding, Azure doesn't support job submission through YARN. It doesn't expose all YARN configs (public facing IP). They recommend "Livy" to submit spark jobs to HDInsight cluster [0]. Any recommendations around integrating it with Livy ?
Thanks, Pankil [0]: https://blogs.msdn.microsoft.com/azuredatalake/2017/01/06/spark-job-submission-on-hdinsight-101/ On Fri, Apr 28, 2017 at 4:24 PM, Donald Szeto <[email protected]> wrote: > Hi Pankil, > > Looks like HDInsight runs Spark on YARN. You would need to point PIO to > the right Hadoop config files, and specify the correct master URL (yarn) > when you run "pio train". > > Regards, > Donald > > On Fri, Apr 28, 2017 at 12:12 PM Pankil Doshi <[email protected]> wrote: > >> Hi, >> >> Has anyone tried running prediction.io with Azure specifically using >> Azure's HDInsight cluster for spark ? >> >> I understand it has capability of pointing to remote spark cluster. But >> trying to figure out what is involved in making it work on HDInsight >> cluster? Can we leverage livy server to submit spark job? >> >> Any pointers would be really helpful. >> >> Thanks, >> Pankil >> >
