dongjoon-hyun opened a new pull request, #59712:
URL: https://github.com/apache/spark/pull/59712
### What changes were proposed in this pull request?
This PR aims to make `TaskSchedulerImpl.stop` discard the pending tasks of
`abortTimer` without waiting, by calling `shutdownNow()` instead of
`ThreadUtils.shutdown`.
```diff
ThreadUtils.shutdown(starvationTimer)
- ThreadUtils.shutdown(abortTimer)
+ // Discard pending abort tasks, delayed by
`UNSCHEDULABLE_TASKSET_TIMEOUT`, without waiting.
+ abortTimer.shutdownNow()
```
When excludeOnFailure is enabled and a task set becomes unschedulable
because all executors are excluded, `TaskSchedulerImpl` puts a one-shot task
into `abortTimer` with the delay of
`spark.scheduler.excludeOnFailure.unschedulableTaskSetTimeout` (default:
`120s`). The task is not cancelled when the task set becomes schedulable again,
so it stays in the queue until the delay elapses.
`DAGScheduler.stop` stops its `messageScheduler`, which also holds one-shot
delayed tasks, with `shutdownNow()` in the same way. If an abort task happens
to be running, `shutdownNow()` interrupts it, but the task only acquires the
scheduler lock and posts an event to the `DAGScheduler`, and neither is
interruptible.
### Why are the changes needed?
Since Apache Spark
[v4.0.0](https://github.com/apache/spark/releases/tag/v4.0.0) (2025-05-19),
`SparkContext.stop` can take up to 30 seconds longer because
`TaskSchedulerImpl.stop` waits for such a pending abort task.
Previously, `abortTimer` was a `java.util.Timer`, and `stop` called
`abortTimer.cancel()`, which discards all scheduled tasks without waiting.
SPARK-46895 (apache/spark#44718) replaced it with a
`ScheduledThreadPoolExecutor` and `ThreadUtils.shutdown(abortTimer)`. However,
a `ScheduledThreadPoolExecutor` still executes the existing delayed tasks after
`shutdown()` by default, so `ThreadUtils.shutdown` waits for its whole
30-second grace period.
The existing test cases are affected too. Before this PR, `afterEach` of
`TaskSchedulerImplSuite` waited 70 seconds in total for the pending abort tasks
of the following test cases.
- `SPARK-22148 try to acquire a new executor when task is unschedulable with
1 executor`: 10 seconds until its 10-second abort task runs.
- `SPARK-22148 abort timer should clear unschedulableTaskSetToExpiryTime for
all TaskSets`: 60 seconds because `TaskSchedulerImpl.stop` is called twice,
directly and via `DAGScheduler.stop`.
### Does this PR introduce _any_ user-facing change?
Yes, `SparkContext.stop` no longer waits up to 30 seconds for the pending
abort timer tasks. This restores the behavior of Apache Spark 3.5. It only
matters when excludeOnFailure is enabled and a task set became unschedulable
within `spark.scheduler.excludeOnFailure.unschedulableTaskSetTimeout` before
stopping.
### How was this patch tested?
Pass the CIs with the newly added test case.
Without this fix, the new test case fails because `stop` waits for 30
seconds.
```
[info] - SPARK-60474: stop should not wait for pending abort timer tasks ***
FAILED *** (30 seconds, 60 milliseconds)
[info] 30005183833 was not less than 10000000000
(TaskSchedulerImplSuite.scala:1173)
```
With this fix, `TaskSchedulerImplSuite` passes in about 7 seconds. Without
this fix, the existing test cases alone took about 77 seconds.
```
$ build/sbt "core/testOnly org.apache.spark.scheduler.TaskSchedulerImplSuite"
...
[info] - SPARK-60474: stop should not wait for pending abort timer tasks (27
milliseconds)
...
[info] Run completed in 6 seconds, 739 milliseconds.
[info] Total number of tests run: 122
[info] Tests: succeeded 122, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
