[
https://issues.apache.org/jira/browse/SPARK-26957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-26957:
----------------------------------
Affects Version/s: (was: 3.0.0)
3.1.0
> Add config properties to configure the default scheduler pool priorities
> ------------------------------------------------------------------------
>
> Key: SPARK-26957
> URL: https://issues.apache.org/jira/browse/SPARK-26957
> Project: Spark
> Issue Type: Improvement
> Components: Scheduler
> Affects Versions: 3.1.0
> Reporter: Dave DeCaprio
> Priority: Minor
>
> Currently, it is possible to dynamically create new scheduler pools in Spark
> just by setting {{spark.scheduler.pool.}} to a new value.
> We use this capability to create separate pools for different projects that
> run jobs in the same long-lived driver application. Each project gets its own
> pool, and within the pool jobs are executed in a FIFO manner.
> This setup works well, except that we also have a low priority queue for
> background tasks. We would prefer for all of the dynamic pools to have a
> higher priority than this background queue.
> We can accomplish this by hardcoding the project queue names in a
> spark_pools.xml config file and setting their priority to 100.
> Unfortunately, there is no way to set the priority for dynamically created
> pools. They are all hardcoded to 1. It would be nice if there were
> configuration settings to change this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
