I think that is Scheduling Within an Application, and he asked across apps. Actually spark standalone supports two ways of scheduling both are FIFO type. http://spark.incubator.apache.org/docs/latest/spark-standalone.html
One is spread out mode and the other is use as fewer node as possible [1] 1. https://github.com/apache/incubator-spark/blob/master/core/src/main/scala/org/apache/spark/deploy/master/Master.scala#L383 On Tue, Nov 19, 2013 at 9:02 PM, Mark Hamstra <[email protected]> wrote: >> >> According to the documentation, spark standalone currently only supports a FIFO scheduling system. > > > That's not true. > > [sorry for the prior misfire] > > > > On Tue, Nov 19, 2013 at 7:30 AM, Mark Hamstra <[email protected]> wrote: >> >> >> >> >> On Tue, Nov 19, 2013 at 6:50 AM, Yadid Ayzenberg <[email protected]> wrote: >>> >>> Hi all, >>> >>> According to the documentation, spark standalone currently only supports a FIFO scheduling system. >>> I understand its possible to limit the number of cores a job uses by setting spark.cores.max. >>> When running a job, will spark try using the max number of cores on each machine until it reaches the set limit, or will it do this round robin style - utilize a single core on each machine - if its already used a core on all of the slaves and the limit has not been reached, spark will utilize an additional core on each machine and so on. >>> >>> I think the latter make more sense, but I want to be sure that is the case. >>> >>> Thanks, >>> Yadid >>> >> > -- s
