SwechchhaSinha opened a new pull request, #58817:
URL: https://github.com/apache/spark/pull/58817
### What changes were proposed in this pull request?
This backports #56957 to branch-3.5.
This PR makes the order in which executor offers are visited during task
scheduling pluggable, via a new `TaskAssignmentStrategy` trait selected by the
new `spark.scheduler.taskAssignmentStrategy` config. Four strategies are
provided:
- `roundrobin` (default): shuffles the offers once per resource-offer
round, preserving the historical default behavior.
- `binpack`: concentrates a task set onto as few executors as possible by
visiting the executor with the fewest live free cores first.
- `balance`: spreads a task set as evenly as possible across executors by
always trying the executor with the most live free cores next.
- `none`: keeps the offers in their given order.
The strategy is created and prepared once per `resourceOffers` round and
reset at the start of every `resourceOfferSingleTaskSet` pass; strategies that
order by free cores read the live `availableCpus` array on each reset.
### Why are the changes needed?
The scheduler previously hard-coded a single offer ordering (a shuffle).
Different workloads benefit from different task placement: bin-packing helps
dynamic allocation scale down by concentrating work so idle executors can be
reclaimed,
while balancing favors parallelism by spreading work out.
### Does this PR introduce any user-facing change?
Yes. A new config `spark.scheduler.taskAssignmentStrategy` (default
`roundrobin`, which matches the previous behavior).
### How was this patch tested?
New unit tests in `TaskSchedulerImplSuite` covering binpack/balance
placement, locality preference, unknown-strategy rejection, barrier task sets,
multiple task sets in one round, and custom resources. `TaskSchedulerImplSuite`
passes (82
tests).
### Was this patch authored or co-authored using generative AI tooling?
Co-authored by: Claude Code (Claude Opus 4.8)
Notes on the backport
- Title kept identical to the 4.0 PR (#56957); the backport + the
referenced main PR are called out in the first line of the description.
- No work item / @W- tag included, per your instruction (this is Apache
Spark OSS).
- 3.5-specific adaptations baked into the commit: Array[Int] instead of
Array[BigDecimal] (3.5 has no fractional task CPUs), dropped the
ConfigBindingPolicy call (absent in 3.5), and version strings set to 3.5.10.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
