Christian Neuenroth created SPARK-58126:
-------------------------------------------
Summary: Support a pluggable SchedulingAlgorithm for the scheduler
root pool via configuration
Key: SPARK-58126
URL: https://issues.apache.org/jira/browse/SPARK-58126
Project: Spark
Issue Type: New Feature
Components: Scheduler
Affects Versions: 5.0.0
Reporter: Christian Neuenroth
Currently the algorithm that orders the scheduler pools (FAIR mode) or task sets
(FIFO mode) is hard-wired: Pool selects either FairSchedulingAlgorithm or
FIFOSchedulingAlgorithm based solely on spark.scheduler.mode. There is no
supported way to customize the ordering — e.g. to implement priority-based
pool ordering, deadline-aware ordering, or business-specific SLAs — without
patching Spark.
SchedulingAlgorithm is a small, well-defined trait
(comparator(s1: Schedulable, s2: Schedulable): Boolean), which makes it a
natural extension point. The top-level pool ordering is governed by the
comparator installed on the root pool, and the root pool is created by
TaskSchedulerImpl, where the SparkConf is available.
Proposal
--------
Introduce a configuration property:
spark.scheduler.rootPool.algorithm.class
When set to the fully qualified name of a class implementing
org.apache.spark.scheduler.SchedulingAlgorithm (with a no-arg constructor),
TaskSchedulerImpl instantiates it (reflectively, via Utils.classForName) and
uses it to order the root pool. When unset, behavior is unchanged: the
algorithm is derived from spark.scheduler.mode (FAIR/FIFO).
To make this a supported extension point, SchedulingAlgorithm and Schedulable
are annotated @DeveloperApi. Schedulable exposes only the read-only scheduling
properties (weight, minShare, runningTasks, priority, stageId, name,
schedulingMode) as public API; structural/mutating members remain
private[spark].
This mirrors the existing pluggable-class pattern used for spark.serializer and
spark.shuffle.manager. It is also the only practical customization path on
managed platforms (Databricks, EMR) where the user does not create the
SparkContext and therefore cannot call a runtime setter.
Scope
-----
- Additive, opt-in, no behavior change when the property is unset.
- Only the root pool ordering is configurable in this change. Per-pool custom
algorithms (via the fair scheduler XML) could be a follow-up.
An alternative runtime API (SparkContext.setRootPoolSchedulingAlgorithm) was
prototyped but rejected in favor of the config-based approach, which is more
idiomatic and works on managed platforms.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
