Actually it is not documented anywhere, maybe we can do that. Was having tough time thinking how to implement this in Spark-743.
On Tue, Nov 19, 2013 at 9:33 PM, Yadid Ayzenberg <[email protected]>wrote: > My bad - I should have stated that up front. I guess it was kind of > implicit within my question. > > Thanks for your help, > > Yadid > > > > On 11/19/13 10:59 AM, Mark Hamstra wrote: > > Ah, sorry -- misunderstood the question. > > > On Nov 19, 2013, at 7:48 AM, Prashant Sharma <[email protected]> wrote: > > I think that is Scheduling Within an Application, and he asked across > apps. Actually spark standalone supports two ways of scheduling both are > FIFO type. > http://spark.incubator.apache.org/docs/latest/spark-standalone.html > > One is spread out mode and the other is use as fewer node as possible > [1] > > 1. > https://github.com/apache/incubator-spark/blob/master/core/src/main/scala/org/apache/spark/deploy/master/Master.scala#L383 > > > > > On Tue, Nov 19, 2013 at 9:02 PM, Mark Hamstra <[email protected]> > wrote: > >> > >> According to the documentation, spark standalone currently only > supports a FIFO scheduling system. > > > > > > That's not true. > > > > [sorry for the prior misfire] > > > > > > > > On Tue, Nov 19, 2013 at 7:30 AM, Mark Hamstra <[email protected]> > wrote: > >> > >> > >> > >> > >> On Tue, Nov 19, 2013 at 6:50 AM, Yadid Ayzenberg <[email protected]> > wrote: > >>> > >>> Hi all, > >>> > >>> According to the documentation, spark standalone currently only > supports a FIFO scheduling system. > >>> I understand its possible to limit the number of cores a job uses by > setting spark.cores.max. > >>> When running a job, will spark try using the max number of cores on > each machine until it reaches the set limit, or will it do this round robin > style - utilize a single core on each machine - if its already used a core > on all of the slaves and the limit has not been reached, spark will utilize > an additional core on each machine and so on. > >>> > >>> I think the latter make more sense, but I want to be sure that is the > case. > >>> > >>> Thanks, > >>> Yadid > >>> > >> > > > > > > -- > s > > > -- s
