My colleague oleg has actually got something better, with spark hooked up to the Tez dataflow engine, so giving dynamic node flexing during a run, automatically, including re-arranging bits of the flow and reacting to failures.
https://issues.apache.org/jira/browse/SPARK-3561 The spark team don't want it though; it's a disagreement about where plugin points go (if any). We've had that in hadoop core, and tend to take the path of "define the extensions points strictly (filesystem, shuffle, scheduler, ...)", write the reference tests, then let anyone else implementing them deal with the compatibility problems. Hopefully the Spark team will choose to do something similar. Anyway: the source for that patch is available —why not have a play with it and see what you think? On 13 January 2015 at 00:58, Yong Feng <[email protected]> wrote: > Hi Slider team > > I also played with "Spark/SparkR on Yarn" recently, and think we even could > run Spark on Slider. After that we could enhance Slider to have API scale > out/in Spark cluster horizontally or vertically. It is a little bit > overlapped with the enhancement on YarnClusterScheduler of Spark. The > Slider also may not have enough knowledge of Spark workload to make > allocate request with data locality. However it will be a more general > solution. After that Slider actually is a meta-app manager on Yarn. It will > be similar as the Marathon in Mesos. > > Does it make sense? or anyone else has similar idea to run Spark on Slider? > Does the data locality matter for Spark? > > Thanks, > > Yong > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
