dongjoon-hyun opened a new pull request, #59126:
URL: https://github.com/apache/spark/pull/59126
### What changes were proposed in this pull request?
This PR aims to add a new executor roll policy, `ACTIVE_TASKS`, to
`spark.kubernetes.executor.rollPolicy`.
The `ACTIVE_TASKS` policy chooses the executor with the smallest number of
active (currently running) tasks, based on `ExecutorSummary.activeTasks`. If
there is a tie, it chooses the executor with the smallest add-time.
```scala
case ExecutorRollPolicy.ACTIVE_TASKS =>
listWithoutDriver.sortBy(e => (e.activeTasks, e.addTime.getTime))
```
`ACTIVE_TASKS` is not included in the `OUTLIER` policy's dimensions, because
a large number of active tasks does not indicate an unhealthy executor.
### Why are the changes needed?
All the existing policies pick an executor based on its health or history
(GC time, task duration, failed tasks, memory, disk, shuffle write, age, and so
on). None of them considers how costly it is to decommission the executor right
now.
Decommissioning an executor that is running many tasks means waiting for
those tasks to finish. If they are killed because of
`spark.executor.decommission.forceKillTimeout`, they have to be recomputed. The
`ACTIVE_TASKS` policy lets users roll executors while disrupting running
workloads as little as possible.
The tie-breaker makes the choice deterministic when many executors have the
same number of active tasks. This is common, for example when all executors are
fully busy or all are idle. Choosing the oldest executor matches the intent of
executor rolling.
### Does this PR introduce _any_ user-facing change?
Yes. `spark.kubernetes.executor.rollPolicy` accepts a new value,
`ACTIVE_TASKS`. The default policy (`OUTLIER`) does not change, so the behavior
of existing jobs does not change.
### How was this patch tested?
Pass the CIs with the newly added test case in `ExecutorRollPluginSuite`.
```
$ build/sbt -Pkubernetes 'kubernetes/testOnly *ExecutorRollPluginSuite'
...
[info] Tests: succeeded 29, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
