Forgot to mention there is this property that controls it called spark.deploy.spreadOut which is by default true. (That is what you observed.)
On Tue, Nov 19, 2013 at 9:18 PM, Prashant Sharma <[email protected]>wrote: > I think that is Scheduling Within an Application, and he asked across > apps. Actually spark standalone supports two ways of scheduling both are > FIFO type. > http://spark.incubator.apache.org/docs/latest/spark-standalone.html > > One is spread out mode and the other is use as fewer node as possible [1] > > 1. > https://github.com/apache/incubator-spark/blob/master/core/src/main/scala/org/apache/spark/deploy/master/Master.scala#L383 > > > > > > On Tue, Nov 19, 2013 at 9:02 PM, Mark Hamstra <[email protected]> > wrote: > >> > >> According to the documentation, spark standalone currently only > supports a FIFO scheduling system. > > > > > > That's not true. > > > > [sorry for the prior misfire] > > > > > > > > On Tue, Nov 19, 2013 at 7:30 AM, Mark Hamstra <[email protected]> > wrote: > >> > >> > >> > >> > >> On Tue, Nov 19, 2013 at 6:50 AM, Yadid Ayzenberg <[email protected]> > wrote: > >>> > >>> Hi all, > >>> > >>> According to the documentation, spark standalone currently only > supports a FIFO scheduling system. > >>> I understand its possible to limit the number of cores a job uses by > setting spark.cores.max. > >>> When running a job, will spark try using the max number of cores on > each machine until it reaches the set limit, or will it do this round robin > style - utilize a single core on each machine - if its already used a core > on all of the slaves and the limit has not been reached, spark will utilize > an additional core on each machine and so on. > >>> > >>> I think the latter make more sense, but I want to be sure that is the > case. > >>> > >>> Thanks, > >>> Yadid > >>> > >> > > > > > > -- > s > -- s
