[
https://issues.apache.org/jira/browse/SPARK-58126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-58126:
-----------------------------------
Labels: pull-request-available (was: )
> Support a pluggable SchedulingAlgorithm for the scheduler root pool via
> configuration
> -------------------------------------------------------------------------------------
>
> Key: SPARK-58126
> URL: https://issues.apache.org/jira/browse/SPARK-58126
> Project: Spark
> Issue Type: New Feature
> Components: Scheduler
> Affects Versions: 5.0.0
> Reporter: Christian Neuenroth
> Priority: Major
> Labels: pull-request-available
>
> Currently the algorithm that orders the scheduler pools (FAIR mode) or task
> sets
> (FIFO mode) is hard-wired: Pool selects either FairSchedulingAlgorithm or
> FIFOSchedulingAlgorithm based solely on spark.scheduler.mode. There is no
> supported way to customize the ordering — e.g. to implement priority-based
> pool ordering, deadline-aware ordering, or business-specific SLAs — without
> patching Spark.
> SchedulingAlgorithm is a small, well-defined trait
> (comparator(s1: Schedulable, s2: Schedulable): Boolean), which makes it a
> natural extension point. The top-level pool ordering is governed by the
> comparator installed on the root pool, and the root pool is created by
> TaskSchedulerImpl, where the SparkConf is available.
> Proposal
> --------
> Introduce a configuration property:
> spark.scheduler.rootPool.algorithm.class
> When set to the fully qualified name of a class implementing
> org.apache.spark.scheduler.SchedulingAlgorithm (with a no-arg constructor),
> TaskSchedulerImpl instantiates it (reflectively, via Utils.classForName) and
> uses it to order the root pool. When unset, behavior is unchanged: the
> algorithm is derived from spark.scheduler.mode (FAIR/FIFO).
> To make this a supported extension point, SchedulingAlgorithm and Schedulable
> are annotated @DeveloperApi. Schedulable exposes only the read-only scheduling
> properties (weight, minShare, runningTasks, priority, stageId, name,
> schedulingMode) as public API; structural/mutating members remain
> private[spark].
> This mirrors the existing pluggable-class pattern used for spark.serializer
> and
> spark.shuffle.manager. It is also the only practical customization path on
> managed platforms (Databricks, EMR) where the user does not create the
> SparkContext and therefore cannot call a runtime setter.
> Scope
> -----
> - Additive, opt-in, no behavior change when the property is unset.
> - Only the root pool ordering is configurable in this change. Per-pool custom
> algorithms (via the fair scheduler XML) could be a follow-up.
> An alternative runtime API (SparkContext.setRootPoolSchedulingAlgorithm) was
> prototyped but rejected in favor of the config-based approach, which is more
> idiomatic and works on managed platforms.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
