Thanks Steve for the info. I will play with it. Yong
On Tue, Jan 13, 2015 at 10:02 AM, Steve Loughran <[email protected]> wrote: > My colleague oleg has actually got something better, with spark hooked up > to the Tez dataflow engine, so giving dynamic node flexing during a run, > automatically, including re-arranging bits of the flow and reacting to > failures. > > https://issues.apache.org/jira/browse/SPARK-3561 > > The spark team don't want it though; it's a disagreement about where plugin > points go (if any). > > We've had that in hadoop core, and tend to take the path of "define the > extensions points strictly (filesystem, shuffle, scheduler, ...)", write > the reference tests, then let anyone else implementing them deal with the > compatibility problems. Hopefully the Spark team will choose to do > something similar. > > Anyway: the source for that patch is available —why not have a play with it > and see what you think? > > > On 13 January 2015 at 00:58, Yong Feng <[email protected]> wrote: > > > Hi Slider team > > > > I also played with "Spark/SparkR on Yarn" recently, and think we even > could > > run Spark on Slider. After that we could enhance Slider to have API scale > > out/in Spark cluster horizontally or vertically. It is a little bit > > overlapped with the enhancement on YarnClusterScheduler of Spark. The > > Slider also may not have enough knowledge of Spark workload to make > > allocate request with data locality. However it will be a more general > > solution. After that Slider actually is a meta-app manager on Yarn. It > will > > be similar as the Marathon in Mesos. > > > > Does it make sense? or anyone else has similar idea to run Spark on > Slider? > > Does the data locality matter for Spark? > > > > Thanks, > > > > Yong > > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
