Hi Pankil, Looks like HDInsight runs Spark on YARN. You would need to point PIO to the right Hadoop config files, and specify the correct master URL (yarn) when you run "pio train".
Regards, Donald On Fri, Apr 28, 2017 at 12:12 PM Pankil Doshi <[email protected]> wrote: > Hi, > > Has anyone tried running prediction.io with Azure specifically using > Azure's HDInsight cluster for spark ? > > I understand it has capability of pointing to remote spark cluster. But > trying to figure out what is involved in making it work on HDInsight > cluster? Can we leverage livy server to submit spark job? > > Any pointers would be really helpful. > > Thanks, > Pankil >
