Can spark.scheduler.pool be applied globally ?

It seems currently spark.scheduler.pool must be set as localProperties (associate with thread). Any reason why spark.scheduler.pool can not be used globally. My scenario is that I want my thriftserver started with fair scheduler as the default pool without using set command to set the pool. Is

Re: Can spark.scheduler.pool be applied globally ?

I don't understand. If you're using fair scheduling and don't set a pool, the default pool will be used. On Tue, Jan 5, 2016 at 1:57 AM, Jeff Zhang wrote: > > It seems currently spark.scheduler.pool must be set as localProperties > (associate with thread). Any reason why

Re: Can spark.scheduler.pool be applied globally ?

The other way to do it is to build a custom version of Spark where you have changed the value of DEFAULT_SCHEDULING_MODE -- and if you were paying close attention, I accidentally let it slip that that is what I've done. I previously wrote "schedulingMode = DEFAULT_SCHEDULING_MODE -- i.e.

Re: Can spark.scheduler.pool be applied globally ?

Thanks Mark, custom configuration file would be better for me. Changing code will make it affect all the applications, this is too risky for me. On Wed, Jan 6, 2016 at 10:50 AM, Mark Hamstra wrote: > The other way to do it is to build a custom version of Spark where

Re: Can spark.scheduler.pool be applied globally ?

Right, I can override the root pool in configuration file, Thanks Mark. On Wed, Jan 6, 2016 at 8:45 AM, Mark Hamstra wrote: > Just configure with > FAIR in fairscheduler.xml (or > in spark.scheduler.allocation.file if you have over-riden the default name > for the

Re: Can spark.scheduler.pool be applied globally ?

Sorry, I don't make it clearly. What I want is the default pool is fair scheduling. But seems if I want to use fair scheduling now, I have to set spark.scheduler.pool explicitly. On Wed, Jan 6, 2016 at 2:03 AM, Mark Hamstra wrote: > I don't understand. If you're using

Re: Can spark.scheduler.pool be applied globally ?

Just configure with FAIR in fairscheduler.xml (or in spark.scheduler.allocation.file if you have over-riden the default name for the config file.) `buildDefaultPool()` will only build the pool named "default" with the default properties (such as schedulingMode = DEFAULT_SCHEDULING_MODE -- i.e.