> > I am using the Spark ‘fair’ scheduler mode.
What do you mean by this? Fair scheduling mode is not one thing in Spark, but allows for multiple configurations and usages. Presumably, at a minimum you are using SparkConf to set spark.scheduling.mode to "FAIR", but then how are you setting up scheduling pools, how are you allocating jobs to pools, and what scheduling mode are you using within pools? Setting spark.scheduling.mode is a necessary but probably not sufficient condition to effect your desired scheduling policy. On Tue, Mar 31, 2015 at 1:19 PM, asadrao <as...@microsoft.com> wrote: > Hi, I am using the Spark ‘fair’ scheduler mode. I have noticed that if the > first query is a very expensive query (ex: ‘select *’ on a really big data > set) than any subsequent query seem to get blocked. I would have expected > the second query to run in parallel since I am using the ‘fair’ scheduler > mode not the ‘fifo’. I am submitting the query through thrift server. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Using-fair-scheduler-mode-tp22328.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
