uros-b commented on code in PR #56628:
URL: https://github.com/apache/spark/pull/56628#discussion_r3466153642
##########
core/src/main/scala/org/apache/spark/executor/Executor.scala:
##########
@@ -571,10 +571,18 @@ private[spark] class Executor(
try {
logError(log"Executor launch task ${MDC(TASK_NAME,
taskDescription.name)} failed," +
log" reason: ${MDC(REASON, t.getMessage)}")
+ // SPARK-57465: If the thread pool rejected the task because the
executor is shutting
+ // down, report a non-counting failure so the task can be
rescheduled elsewhere.
+ val reason: TaskFailedReason = t match {
+ case _: RejectedExecutionException if executorShutdown.get() =>
Review Comment:
Coverage gap for
`core/src/test/scala/org/apache/spark/executor/ExecutorSuite.scala`: the actual
new mapping logic (RejectedExecutionException && executorShutdown.get() ->
ExecutorShutdownFailure, and the negative path where a non-shutdown rejection
must still become ExceptionFailure) in Executor.launchTask is untested. The
merged SPARK-54087 test (line 619) already mocks threadPool.execute via
reflection and deserializes the captured statusUpdate reason; a ready-made
pattern. Add (a) shutdown-flag-set -> ExecutorShutdownFailure and (b)
flag-unset -> ExceptionFailure. The added TaskSetManagerSuite tests bypass the
Executor path entirely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
