Hi Slider team I also played with "Spark/SparkR on Yarn" recently, and think we even could run Spark on Slider. After that we could enhance Slider to have API scale out/in Spark cluster horizontally or vertically. It is a little bit overlapped with the enhancement on YarnClusterScheduler of Spark. The Slider also may not have enough knowledge of Spark workload to make allocate request with data locality. However it will be a more general solution. After that Slider actually is a meta-app manager on Yarn. It will be similar as the Marathon in Mesos.
Does it make sense? or anyone else has similar idea to run Spark on Slider? Does the data locality matter for Spark? Thanks, Yong
