dongjoon-hyun opened a new pull request, #58171:
URL: https://github.com/apache/spark/pull/58171
### What changes were proposed in this pull request?
This PR adds `TaskSchedulerImpl.hasPipelinedTaskSets`, a query for whether
any live task set belongs to a pipelined group:
```scala
private[spark] def hasPipelinedTaskSets: Boolean = synchronized {
taskSetsByStageIdAndAttempt.values.exists(_.values.exists { tsm =>
!tsm.isZombie && tsm.taskSet.isPipelined
})
}
```
It sits next to `outstandingTasksForOtherWorkInProfile`, the other
pipelined-group helper over the same map, and follows its conventions: taken
under the existing lock, and skipping zombie (superseded) attempts.
This is a spin-off of https://github.com/apache/spark/pull/58054, which
introduces `SparkContext.holdExecutors()` and consumes this method.
### Why are the changes needed?
A pipelined group's shuffle data is transient and lives only on the
executors running the group, so a caller about to disturb those executors --
graceful decommission in #58054 -- needs to know that such a group is running.
No existing API answers that: `TaskSet.isPipelined` is per task set, and
`taskSetsByStageIdAndAttempt` is private to `TaskSchedulerImpl`.
The check is best-effort by design. It reports what the *task* scheduler
currently holds, which is narrower than "a pipelined job is active": it is
false before the group's first task set is submitted, and false again once the
last member's `TaskSetManager` has gone zombie but the `DAGScheduler` has not
yet processed the final completion event.
### Does this PR introduce _any_ user-facing change?
No. `hasPipelinedTaskSets` is `private[spark]` and has no caller in this PR.
### How was this patch tested?
Three new unit tests in `TaskSchedulerImplSuite` cover a live pipelined task
set, the `!isZombie` filter in both directions, and the tail of the best-effort
window.
```
build/sbt 'core/testOnly *TaskSchedulerImplSuite -- -z SPARK-58913'
```
```
[info] Tests: succeeded 3, failed 0, canceled 0, ignored 0, pending 0
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
