Re: pyspark - spark history server

2016-02-05 Thread cs user
person. Thanks! On Fri, Feb 5, 2016 at 2:08 PM, cs user <acldstk...@gmail.com> wrote: > Hi All, > > I'm having trouble getting a job to use the spark history server. We have > a cluster configured with Ambari, if I run the job from one of the nodes > within the Amba

pyspark - spark history server

2016-02-05 Thread cs user
Hi All, I'm having trouble getting a job to use the spark history server. We have a cluster configured with Ambari, if I run the job from one of the nodes within the Ambari configured cluster, everything works fine, the job appears in the spark history server. If I configure a client external to

Re: Autoscaling of Spark YARN cluster

2015-12-14 Thread cs user
Hi Mingyu, I'd be interested in hearing about anything else you find which might meet your needs for this. One way perhaps this could be done would be to use Ambari. Ambari comes with a nice api which you can use to add additional nodes into a cluster:

Spark on yarn vs spark standalone

2015-11-26 Thread cs user
Hi All, Apologies if this question has been asked before. I'd like to know if there are any downsides to running spark over yarn with the --master yarn-cluster option vs having a separate spark standalone cluster to execute jobs? We're looking at installing a hdfs/hadoop cluster with Ambari and