This is an automated email from the ASF dual-hosted git repository.

viirya pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new ff82f3a67e16 [SPARK-60222][CORE] Keep task completion listeners on the 
completing thread when another thread invokes listeners
ff82f3a67e16 is described below

commit ff82f3a67e16df106ec15a970910950a35cf3112
Author: Liang-Chi Hsieh <[email protected]>
AuthorDate: Sat Oct 31 00:14:50 2026 -0700

    [SPARK-60222][CORE] Keep task completion listeners on the completing thread 
when another thread invokes listeners
    
    ### What changes were proposed in this pull request?
    
    When another thread is invoking task listeners at the moment the task 
completes (e.g. a kill thread running failure listeners because an interrupt 
listener threw, or a non-task thread that called `markTaskFailed` through 
`Utils.tryWithSafeFinallyAndFailureCallbacks`), 
`TaskContextImpl.markTaskCompleted` returns without running the completion 
listeners, and the other thread only drains its own stack, so the completion 
listeners never run. Failure listeners that another thread queues wh [...]
    
    This PR keeps the completion listeners on the completing thread:
    
    - `markTaskCompleted` claims the listener invocation in the same 
synchronized block that sets `completed`. If another thread holds it, 
`markTaskCompleted` registers itself as the next invoker and waits; the holder 
hands the invocation over directly when it is done, so no third thread can take 
it in between.
    - When a listener throws, the failure listeners run nested in the current 
invocation instead of clearing and reclaiming `listenerInvocationThread`, which 
left a window in which another thread could take over the listeners.
    - Failure listeners that another thread queues while `markTaskCompleted` 
runs the completion listeners run on the completing thread before the 
completion listeners that have not run yet, so failure cleanup (e.g. a state 
store `abort()`) runs before the completion listener that releases the store if 
that one has not run yet, also after a handover. This is the order in which a 
failing completion listener already runs the failure listeners on master, and 
their failures are likewise repor [...]
    - The release and the handover are in a `finally`, and `markTaskCompleted` 
removes a waiter or holder that it registered if an error leaves it behind.
    - The handover wait keeps interrupts (they are restored once it is done) 
and a failure while logging (it is thrown after the handover with the 
completion listeners' failures, unless one of them is fatal, or reported after 
the failures of the pre-completion listeners if there are any), and logs a 
warning periodically; the first one carries the holder's stack trace, taken 
outside the TaskContext lock once only, and is skipped if the holder has handed 
over meanwhile. The interval is sett [...]
    - A failure listener invocation that a failing listener starts runs nested, 
and an error from it that `Utils.isFatalError` does not count as fatal, e.g. a 
`LinkageError`, is reported with that listener's failure instead of ending the 
listener loop. A fatal error, e.g. an `OutOfMemoryError`, propagates.
    - `getMessage` is listener code: if it fails for a listener's exception, 
the `TaskCompletionListenerException` reports that failure in place of the 
exception, so that the other listeners' failures are still reported. This 
applies to the completion, failure and interrupt listeners.
    - The StreamingShuffleSuite test that documented the skipped completion 
listener now checks that it runs on the completing thread, and the comments in 
`ErrorNotifier` and `StreamingShuffleReader` are updated accordingly.
    
    This was split out of #59332 during its review, since it changes listener 
behavior for every task.
    
    ### Why are the changes needed?
    
    Completion listeners do the task's cleanup (closing readers, releasing 
state stores, freeing task-managed resources). When they are skipped, those 
resources leak; when they run late on another thread, they run after the 
executor has already released the task's memory. Failure listeners that are 
dropped skip failure cleanup such as aborting a state store update.
    
    Note on the wait: it has no time bound, so if the holder's listeners block, 
the completing task waits too, and it now waits before the executor releases 
the task's block locks and memory. On master, the task thread could release 
them, but not report its result, before it blocked on the `TaskRunner` lock 
that a kill thread holds while it runs `Task.kill`. So a failure listener on 
another thread, e.g. a kill thread, must not wait for the task's block locks or 
memory. When the holder is  [...]
    
    Out of scope: listeners added after `markTaskCompleted` has finished, while 
another thread holds the invocation, are still dropped unless that thread runs 
listeners of the same kind, and so are post-status-update listeners if another 
thread holds the invocation when `invokePostStatusUpdateListeners` runs. 
Running them would mean running them on another thread, without the task's 
`TaskContext`, after the executor has released the task's block locks and 
memory.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No API change. In the cases above, completion and failure listeners that 
were skipped now run, and `markTaskCompleted` waits for the other thread to 
finish its listeners first.
    
    ### How was this patch tested?
    
    New tests in `TaskContextSuite`:
    - a listener added from another thread as completion starts does not take 
over the listeners;
    - a failing listener does not let another thread take over the listeners;
    - completion waits for another thread invoking listeners (a kill whose 
interrupt listener throws) and runs the listeners on the completing thread;
    - a listener on the invoking thread that completes the task runs the 
completion listeners nested, and that thread keeps the invocation;
    - a failure listener left to the completing thread runs before its next 
completion listener, without and with a handover, and its failure is reported 
as its own;
    - an interrupt does not stop the handover wait and is kept;
    - the handover wait warns with the holder's name and, on the first warning 
only, its stack trace, and does not warn about a holder that has handed over;
    - a failing handover warning does not end the wait: the completing thread 
runs the listeners once the holder has handed over, then throws the failure 
(with an appender that does not ignore its exceptions);
    - a failing handover warning is reported after the failure of a 
pre-completion listener, and with the failure of a completion listener, unless 
the completion listener invocation throws an `OutOfMemoryError`, which then 
propagates;
    - a failure of the `getMessage` of a listener's exception is reported in 
place of that exception, together with the other listeners' failures, for 
failure listeners started by a failing pre-completion or completion listener or 
left to the completing thread, for pre-completion listeners and for interrupt 
listeners;
    - a `LinkageError` (from a `getMessage` whose failure fails in `getMessage` 
too) does not end the pre-completion, completion or left failure listener loop, 
and is reported with the failure of a pre-completion listener;
    - an error that escapes the listener loop (an `OutOfMemoryError` from such 
a `getMessage`) still releases the invocation, and propagates instead of 
becoming a suppressed exception of the listener whose failure started it;
    - completion and failure listeners added after completion are not run by a 
thread invoking other listeners.
    
    Updated tests: "background markTaskFailed races with task thread 
markTaskCompleted" in `StreamingShuffleSuite`, and "SPARK-56330: task 
completion during interrupt listener execution" in `TaskContextSuite`, which 
now completes the task on a thread that it joins with a timeout. The threads 
these tests start record what they throw, and the test fails if one did.
    
    Each behavior that these tests cover was checked by reverting it alone and 
confirming that a test fails. Not covered are the allocations moved above the 
claim, the cleanup in `markTaskCompleted`'s `finally`, which only an error such 
as an `OutOfMemoryError` before `invokeListeners` starts can reach, the 
handover when an error escapes the holder's listener loop (the loop of a 
failure listener invocation does not start a nested one), and the handling of 
an `InterruptedException` while w [...]
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (Claude Opus 5.5)
    
    This pull request and its description were written by Isaac.
    
    Closes #59663 from viirya/SPARK-60222-listener-handover.
    
    Authored-by: Liang-Chi Hsieh <[email protected]>
    Signed-off-by: Liang-Chi Hsieh <[email protected]>
---
 .../scala/org/apache/spark/TaskContextImpl.scala   | 414 ++++++++++--
 .../org/apache/spark/api/python/PythonRunner.scala |   3 +-
 .../shuffle/streaming/StreamingShuffleReader.scala |   4 +-
 .../org/apache/spark/util/ErrorNotifier.scala      |   7 +-
 .../apache/spark/scheduler/TaskContextSuite.scala  | 751 ++++++++++++++++++++-
 .../shuffle/streaming/StreamingShuffleSuite.scala  | 109 +--
 6 files changed, 1146 insertions(+), 142 deletions(-)

diff --git a/core/src/main/scala/org/apache/spark/TaskContextImpl.scala 
b/core/src/main/scala/org/apache/spark/TaskContextImpl.scala
index 575d0e6f7512..45e8808812b7 100644
--- a/core/src/main/scala/org/apache/spark/TaskContextImpl.scala
+++ b/core/src/main/scala/org/apache/spark/TaskContextImpl.scala
@@ -19,6 +19,7 @@ package org.apache.spark
 
 import java.io.Closeable
 import java.util.{Properties, Stack}
+import java.util.concurrent.TimeUnit
 import javax.annotation.concurrent.GuardedBy
 
 import scala.collection.mutable.ArrayBuffer
@@ -27,7 +28,7 @@ import scala.util.control.NonFatal
 
 import org.apache.spark.executor.TaskMetrics
 import org.apache.spark.internal.{config, Logging}
-import org.apache.spark.internal.LogKeys.LISTENER
+import org.apache.spark.internal.LogKeys.{LISTENER, TASK_ATTEMPT_ID, 
THREAD_NAME, TOTAL_TIME}
 import org.apache.spark.memory.TaskMemoryManager
 import org.apache.spark.metrics.MetricsSystem
 import org.apache.spark.metrics.source.Source
@@ -63,6 +64,8 @@ private[spark] class TaskContextImpl(
   extends TaskContext
   with Logging {
 
+  import TaskContextImpl._
+
   /**
    * List of callback functions to execute when the task completes.
    *
@@ -94,9 +97,21 @@ private[spark] class TaskContextImpl(
   /**
    * The thread currently executing task completion or failure listeners, if 
any.
    *
-   * `invokeListeners()` uses this to ensure listeners are called sequentially.
+   * `invokeListeners()` uses this to ensure listeners are called 
sequentially. Protected by the
+   * TaskContext lock.
+   */
+  @transient private var listenerInvocationThread: Option[Thread] = None
+
+  /**
+   * A thread waiting in `markTaskCompleted()` for another thread to stop 
invoking listeners. That
+   * thread hands the listener invocation directly to this one, so that no 
third thread can take
+   * it over in between. Protected by the TaskContext lock.
    */
-  @transient @volatile private var listenerInvocationThread: Option[Thread] = 
None
+  @transient private var nextListenerInvocationThread: Option[Thread] = None
+
+  /** How often `markTaskCompleted()` logs while it waits for another thread 
invoking listeners. */
+  @transient @volatile private[spark] var listenerHandoverWarnIntervalMs: Long 
=
+    DEFAULT_LISTENER_HANDOVER_WARN_INTERVAL_MS
 
   // If defined, the corresponding task has been killed and this option 
contains the reason.
   @volatile private var reasonIfKilled: Option[String] = None
@@ -125,9 +140,12 @@ private[spark] class TaskContextImpl(
 
   override def addTaskCompletionListener(listener: TaskCompletionListener): 
this.type = {
     val needToCallListener = synchronized {
-      // If there is already a thread invoking listeners, adding the new 
listener to
-      // `onCompleteCallbacks` will cause that thread to execute the new 
listener, and the call to
-      // `invokeTaskCompletionListeners()` below will be a no-op.
+      // If there is already a thread invoking listeners, the call to
+      // `invokeTaskCompletionListeners()` below will be a no-op. If that 
thread runs completion
+      // listeners (in `markTaskCompleted()`, or since it added one after 
completion), or hands the
+      // invocation over to a thread waiting to do so, the new listener runs 
there. Otherwise it is
+      // not run until a completion listener is added while no thread is 
invoking listeners (see
+      // `invokeListeners()`).
       //
       // If there is no such thread, the call to 
`invokeTaskCompletionListeners()` below will
       // execute all listeners, including the new listener.
@@ -135,7 +153,7 @@ private[spark] class TaskContextImpl(
       completed
     }
     if (needToCallListener) {
-      invokeTaskCompletionListeners(None)
+      invokeTaskCompletionListeners(None, Claim)
     }
     this
   }
@@ -162,7 +180,7 @@ private[spark] class TaskContextImpl(
     synchronized {
       onFailureCallbacks.push(listener)
       failureCauseOpt
-    }.foreach(invokeTaskFailureListeners)
+    }.foreach(invokeTaskFailureListeners(_, Claim))
     this
   }
 
@@ -189,7 +207,7 @@ private[spark] class TaskContextImpl(
   }
 
   private[spark] def invokePostStatusUpdateListeners(): Unit = {
-    invokeListeners(onPostStatusUpdateCallbacks, "PostStatusUpdateListener", 
None) {
+    invokeListeners(onPostStatusUpdateCallbacks, "PostStatusUpdateListener", 
None, Claim) {
       _.onStatusUpdateSent(this)
     }
   }
@@ -199,11 +217,19 @@ private[spark] class TaskContextImpl(
   }
 
   private[spark] override def markTaskFailed(error: Throwable): Unit = {
+    markTaskFailed(error, Claim)
+  }
+
+  /**
+   * @param mode `Nested` if this thread is already invoking listeners, i.e. a 
listener failed. The
+   *             failure listeners then run as part of that invocation, which 
this thread keeps.
+   */
+  private def markTaskFailed(error: Throwable, mode: InvocationMode): Unit = {
     synchronized {
       if (failureCauseOpt.isDefined) return
       failureCauseOpt = Some(error)
     }
-    invokeTaskFailureListeners(error)
+    invokeTaskFailureListeners(error, mode)
   }
 
   private[spark] override def markTaskCompleted(error: Option[Throwable]): 
Unit = {
@@ -218,30 +244,97 @@ private[spark] class TaskContextImpl(
     // writer that adds listeners lazily while it pulls its input. And since 
the completion
     // listeners only run after `completed` is set, they run after every 
pre-completion listener,
     // even if another thread is invoking listeners at this point.
+    //
+    // The completion listeners then run on this thread too, and this method 
returns only after
+    // they have run. If it returned while another thread is invoking 
listeners (e.g. a kill thread
+    // running failure listeners), that thread would not run them, and the 
task would go on to
+    // release its memory with them pending. So the listener invocation is 
claimed together with
+    // setting `completed`, which also keeps a thread that adds a completion 
listener at that
+    // moment from running them; if another thread holds it, that thread hands 
it over to this one
+    // when done. Failure listeners that another thread leaves to this 
invocation run before the
+    // next completion listener (see `invokeListeners()`). A completion 
listener added after this
+    // method has run the others runs on the thread adding it, unless another 
thread is invoking
+    // listeners at that moment: that thread runs it only if it is running 
completion listeners
+    // (see `invokeListeners()`).
+    //
+    // The wait for the other thread has no time bound: if its listeners 
block, so does this task.
+    // It logs a warning periodically, the first one with that thread's stack 
trace. The task waits
+    // here before it releases its block locks and memory (the executor does 
that once `Task.run`
+    // returns), so a failure listener on another thread, e.g. a kill thread, 
must not wait for
+    // either. Nothing bounds a wait for a kill thread: with 
`spark.task.reaper.enabled`, every
+    // kill runs on a reaper thread, which checks 
`spark.task.reaper.killTimeout` only after its
+    // kill call returns. If another thread holds the invocation and the task 
is killed, the reaper
+    // with a positive kill timeout bounds the wait by killing the executor 
JVM, except in local
+    // mode, where it only logs an error.
+    val currentThread = Thread.currentThread()
+    var mode: InvocationMode = Claimed
+    var awaitHandover = false
+    // Whether this call registered this thread as the listener invocation 
thread or as the next
+    // one, so that it must make sure not to leave it registered.
+    var registered = false
     val preCompletionExceptions = new ArrayBuffer[Throwable](2)
     def nextPreCompletionListener(): Option[TaskCompletionListener] = 
synchronized {
       if (onPreCompleteCallbacks.empty()) {
         // Set together with finding no more pre-completion listeners, so that 
one added later
-        // is handed to `addTaskCompletionListener` instead of being left in 
the stack.
+        // is handed to `addTaskCompletionListener` instead of being left in 
the stack. The
+        // listener invocation is claimed in the same step (see above).
         completed = true
+        if (listenerInvocationThread.isEmpty) {
+          listenerInvocationThread = Some(currentThread)
+          registered = true
+        } else if (listenerInvocationThread.contains(currentThread)) {
+          // Called from a listener on this thread: run the completion 
listeners as part of that
+          // invocation.
+          mode = Nested
+        } else {
+          nextListenerInvocationThread = Some(currentThread)
+          awaitHandover = true
+          registered = true
+        }
         None
       } else {
         Some(onPreCompleteCallbacks.pop())
       }
     }
     var listener: Option[TaskCompletionListener] = None
-    while ({listener = nextPreCompletionListener(); listener.nonEmpty}) {
-      invokeTaskPreCompletionListener(listener.get, preCompletionExceptions)
-    }
-
     try {
-      invokeTaskCompletionListeners(error)
-    } catch {
-      case e: TaskCompletionListenerException if 
preCompletionExceptions.nonEmpty =>
-        // Merge the completion listeners' own failures, not the exception 
that wraps them. Any
-        // other error comes from invokeListeners itself (e.g. an 
OutOfMemoryError) and is left
-        // to propagate, so that the executor still sees a fatal one.
-        preCompletionExceptions ++= e.getSuppressed.toSeq
+      while ({listener = nextPreCompletionListener(); listener.nonEmpty}) {
+        invokeTaskPreCompletionListener(listener.get, preCompletionExceptions)
+      }
+      try {
+        val warningFailure =
+          if (awaitHandover) awaitListenerInvocationHandover(currentThread) 
else None
+        try {
+          invokeTaskCompletionListeners(error, mode)
+        } catch {
+          // The failure to warn came after the pre-completion listeners' 
failures, but before
+          // those of the completion listeners. A fatal error propagates 
instead.
+          case e if !Utils.isFatalError(e) && warningFailure.exists(_ ne e) =>
+            warningFailure.get.addSuppressed(e)
+        }
+        warningFailure.foreach(throw _)
+      } catch {
+        // Merge the completion listeners' own failures, not the exception 
that wraps them, and
+        // any other failure that is not fatal (see `Utils.isFatalError()`), 
e.g. of the warning,
+        // so that the pre-completion listeners' failures are reported first. 
A fatal error (e.g.
+        // an OutOfMemoryError) propagates and ends the task's listeners, as 
it ends the executor.
+        case e: TaskCompletionListenerException if 
preCompletionExceptions.nonEmpty =>
+          preCompletionExceptions ++= e.getSuppressed.toSeq
+        case e if !Utils.isFatalError(e) && preCompletionExceptions.nonEmpty =>
+          preCompletionExceptions += e
+      }
+    } finally {
+      if (registered) synchronized {
+        // `invokeListeners()` releases the invocation itself once it has 
started; this covers an
+        // error before that, e.g. an OutOfMemoryError. A stale waiter would 
get the invocation
+        // handed to it, and a stale holder would keep every other thread from 
running listeners.
+        if (nextListenerInvocationThread.contains(currentThread)) {
+          nextListenerInvocationThread = None
+        }
+        if (listenerInvocationThread.contains(currentThread)) {
+          releaseListenerInvocation()
+        }
+      }
     }
     if (preCompletionExceptions.nonEmpty) {
       throw newTaskCompletionListenerException(preCompletionExceptions.toSeq, 
error)
@@ -266,7 +359,9 @@ private[spark] class TaskContextImpl(
         try {
           markTaskFailed(e)
         } catch {
-          case t: Throwable => e.addSuppressed(t)
+          // Failures of the failure listeners. A fatal error (see 
`Utils.isFatalError()`, e.g. an
+          // OutOfMemoryError) propagates and ends the task's listeners, as it 
ends the executor.
+          case t if !Utils.isFatalError(t) => e.addSuppressed(t)
         }
         listenerExceptions += e
         // Best effort: a failure to log, e.g. from an appender that does not 
ignore exceptions,
@@ -283,32 +378,140 @@ private[spark] class TaskContextImpl(
   private def newTaskCompletionListenerException(
       listenerExceptions: Seq[Throwable],
       error: Option[Throwable]): TaskCompletionListenerException = {
-    val exception = new 
TaskCompletionListenerException(listenerExceptions.map(_.getMessage), error)
-    listenerExceptions.foreach(exception.addSuppressed)
+    // `getMessage` is listener code. Report its failure in place of the 
exception it fails for:
+    // `printStackTrace`, e.g. in `Utils.exceptionString`, would fail on that 
exception as well.
+    val (reported, messages) = listenerExceptions.map { e =>
+      try {
+        (e, e.getMessage)
+      } catch {
+        case t if !Utils.isFatalError(t) => (t, t.getMessage)
+      }
+    }.unzip
+    val exception = new TaskCompletionListenerException(messages, error)
+    reported.foreach(exception.addSuppressed)
     exception
   }
 
-  private def invokeTaskCompletionListeners(error: Option[Throwable]): Unit = {
+  /**
+   * Waits until the thread invoking listeners hands the invocation over to 
`currentThread`, which
+   * has registered itself as `nextListenerInvocationThread`.
+   *
+   * Interrupts do not stop the wait, and are passed on once it is done. The 
wait of
+   * `PipelinedWriterTask.awaitExit()` in `BasePythonRunner` drops them 
instead, because it is a
+   * listener itself and the listeners right after it must still be able to 
clean up. This wait
+   * is not a listener but part of `markTaskCompleted()`, so it leaves the 
interrupt status (e.g.
+   * set by a task kill) as the completion listeners would see it without the 
wait. If the other
+   * thread is a kill thread, the interrupt of that kill does not come during 
the wait: `Task.kill`
+   * interrupts the task thread once `markInterrupted()` has returned, after 
the handover, so with
+   * `interruptThread` it may come while the completion listeners run here.
+   *
+   * A failure to warn (e.g. an OutOfMemoryError while logging) does not end 
the wait either, like
+   * in `PipelinedWriterTask.awaitExit()`: the other thread does not run the 
completion listeners,
+   * and the task would release its memory with them pending. The first such 
failure is returned,
+   * for `markTaskCompleted()` to throw once it has run the completion 
listeners. The wait itself
+   * is not covered, so that a lasting failure does not spin.
+   */
+  private def awaitListenerInvocationHandover(currentThread: Thread): 
Option[Throwable] = {
+    val startNs = System.nanoTime()
+    var interrupted = false
+    var warned = false
+    var handedOver = false
+    var failure: Option[Throwable] = None
+    try {
+      while (!handedOver) {
+        val holder = synchronized {
+          val deadlineNs = System.nanoTime() +
+            TimeUnit.MILLISECONDS.toNanos(math.max(1L, 
listenerHandoverWarnIntervalMs))
+          var remainingNs = deadlineNs - System.nanoTime()
+          while (!listenerInvocationThread.contains(currentThread) && 
remainingNs > 0) {
+            try {
+              wait(math.max(1L, TimeUnit.NANOSECONDS.toMillis(remainingNs)))
+            } catch {
+              case _: InterruptedException => interrupted = true
+            }
+            remainingNs = deadlineNs - System.nanoTime()
+          }
+          handedOver = listenerInvocationThread.contains(currentThread)
+          if (handedOver) None else listenerInvocationThread
+        }
+        // Outside the lock: walking the stack of another thread stops it at a 
safepoint, and
+        // logging may block, which would otherwise hold up every thread using 
this TaskContext.
+        // The holder does not change during the wait, since it hands the 
invocation to this
+        // thread, so its stack trace is taken for the first warning only, 
even if that fails.
+        holder match {
+          case Some(holder) =>
+            val withStackTrace = !warned
+            warned = true
+            try {
+              warnStillWaiting(holder, startNs, withStackTrace)
+            } catch {
+              // E.g. from an appender that blocks interruptibly: an interrupt 
like any other.
+              case _: InterruptedException => interrupted = true
+              case t: Throwable => failure = failure.orElse(Some(t))
+            }
+          case None =>
+        }
+      }
+    } finally {
+      if (interrupted) {
+        currentThread.interrupt()
+      }
+    }
+    failure
+  }
+
+  private def warnStillWaiting(holder: Thread, startNs: Long, withStackTrace: 
Boolean): Unit = {
+    val holderStack = if (withStackTrace) Some(holder.getStackTrace) else None
+    // If the holder has handed over meanwhile, its stack shows whatever it 
does next.
+    if (synchronized(listenerInvocationThread.contains(holder))) {
+      val waitedMs = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startNs)
+      val message = log"Task ${MDC(TASK_ATTEMPT_ID, taskAttemptId)} has been 
waiting " +
+        log"for ${MDC(TOTAL_TIME, waitedMs)} ms to run its completion 
listeners until " +
+        log"thread ${MDC(THREAD_NAME, holder.getName)} finishes running its 
task listeners."
+      holderStack match {
+        case Some(trace) =>
+          val stack = new Exception(s"Stack trace of thread ${holder.getName}")
+          stack.setStackTrace(trace)
+          logWarning(message, stack)
+        case None =>
+          logWarning(message)
+      }
+    }
+  }
+
+  /**
+   * Deregisters this thread as the listener invocation thread, or hands the 
invocation over to a
+   * thread waiting for it in `markTaskCompleted()`. Must hold the TaskContext 
lock.
+   */
+  private def releaseListenerInvocation(): Unit = {
+    listenerInvocationThread = nextListenerInvocationThread
+    nextListenerInvocationThread = None
+    notifyAll()
+  }
+
+  private def invokeTaskCompletionListeners(
+      error: Option[Throwable],
+      mode: InvocationMode): Unit = {
     // It is safe to access the reference to `onCompleteCallbacks` without 
holding the TaskContext
     // lock. `invokeListeners()` acquires the lock before accessing the 
contents.
-    invokeListeners(onCompleteCallbacks, "TaskCompletionListener", error) {
+    invokeListeners(onCompleteCallbacks, "TaskCompletionListener", error, 
mode) {
       _.onTaskCompletion(this)
     }
   }
 
-  private def invokeTaskFailureListeners(error: Throwable): Unit = {
+  private def invokeTaskFailureListeners(error: Throwable, mode: 
InvocationMode): Unit = {
     // It is safe to access the reference to `onFailureCallbacks` without 
holding the TaskContext
     // lock. `invokeListeners()` acquires the lock before accessing the 
contents.
-    invokeListeners(onFailureCallbacks, "TaskFailureListener", Option(error)) {
+    invokeListeners(onFailureCallbacks, "TaskFailureListener", Option(error), 
mode) {
       _.onTaskFailure(this, error)
     }
   }
 
   private def invokeTaskInterruptListeners(reason: String, error: Throwable): 
Unit = {
     // Do not use `invokeListeners()`. That method uses 
`listenerInvocationThread` to serialize
-    // all listener invocations, which would prevent task completion or 
failure listeners from
-    // running if the task completes or fails while executing an interrupt 
listener (causing
-    // resource leaks such as task-managed resources not being freed).
+    // all listener invocations, so if the task completes or fails while an 
interrupt listener
+    // runs, the task completion would wait for the interrupt listeners, and 
the failure listeners
+    // would only run after them.
     //
     // Instead, directly execute the task interrupt listeners with independent 
serialization.
     // Exceptions are collected and thrown as a 
TaskCompletionListenerException so that
@@ -339,17 +542,37 @@ private[spark] class TaskContextImpl(
       }
     }
     if (listenerExceptions.nonEmpty) {
-      val exception = new TaskCompletionListenerException(
-        listenerExceptions.map(_.getMessage).toSeq, Option(error))
-      listenerExceptions.foreach(exception.addSuppressed)
-      throw exception
+      throw newTaskCompletionListenerException(listenerExceptions.toSeq, 
Option(error))
     }
   }
 
+  /**
+   * Invokes and removes the listeners in `listeners`, in reverse order of 
registration.
+   *
+   * Other threads cannot invoke listeners while this thread does, so they 
leave the listeners
+   * they add, or whose invocation they start, meanwhile. Those in `listeners` 
run here like the
+   * others. If this is the completion listener invocation of 
`markTaskCompleted()`, the failure
+   * listeners left to it (the task has failed) run before the completion 
listeners that have
+   * not run yet, e.g. so that a failure listener that aborts a state store 
update runs before
+   * the completion listener that releases the state store, if that one has 
not run yet. This is
+   * the order of a failure listener invocation that a failing completion 
listener starts, and,
+   * likewise, the failures of these failure listeners are reported with those 
of the completion
+   * listeners. Any other invocation does not run the listeners of another 
kind that other threads
+   * left to it, since its thread may be one on which they must not run, e.g. 
a kill thread, or the
+   * task thread once `Task.run` has unset its `TaskContext` and the executor 
has released the
+   * task's block locks and memory. A listener that fails here still starts 
the failure listeners
+   * on this thread (see `runListener`). Unless `mode` is `Nested`, this 
thread releases the
+   * invocation when it is done, or hands it over to a thread waiting for it in
+   * `markTaskCompleted()`.
+   *
+   * Exceptions of the failure listeners left to this invocation are reported 
to the caller of
+   * this method, together with those of `listeners`.
+   */
   private def invokeListeners[T](
       listeners: Stack[T],
       name: String,
-      error: Option[Throwable])(
+      error: Option[Throwable],
+      mode: InvocationMode)(
       callback: T => Unit): Unit = {
     // This method is subject to two constraints:
     //
@@ -361,7 +584,16 @@ private[spark] class TaskContextImpl(
     //
     // We meet these constraints by ensuring there is at most one thread 
invoking listeners at any
     // point in time.
-    synchronized {
+    //
+    // Allocated before the claim below, so that nothing can fail between the 
claim and the `try`
+    // that releases it.
+    //
+    // Whether this is the completion listener invocation of 
`markTaskCompleted()`, the only caller
+    // of `invokeTaskCompletionListeners()` that does not claim the invocation.
+    val completesTask = (listeners eq onCompleteCallbacks) && mode != Claim
+    val listenerExceptions = new ArrayBuffer[Throwable](2)
+    var released = mode == Nested
+    if (mode == Claim) synchronized {
       if (listenerInvocationThread.nonEmpty) {
         // If another thread is already invoking listeners, do nothing.
         return
@@ -373,26 +605,14 @@ private[spark] class TaskContextImpl(
       }
     }
 
-    def getNextListenerOrDeregisterThread(): Option[T] = synchronized {
-      if (listeners.empty()) {
-        // We have executed all listeners that have been added so far. 
Deregister this thread as the
-        // callback invocation thread.
-        listenerInvocationThread = None
-        None
-      } else {
-        Some(listeners.pop())
-      }
-    }
-
-    val listenerExceptions = new ArrayBuffer[Throwable](2)
-    var listenerOption: Option[T] = None
-    while ({listenerOption = getNextListenerOrDeregisterThread(); 
listenerOption.nonEmpty}) {
-      val listener = listenerOption.get
+    def runListener(listener: T): Unit = {
       try {
         callback(listener)
       } catch {
         case e: Throwable =>
-          // A listener failed. Temporarily clear the listenerInvocationThread 
and markTaskFailed.
+          // A listener failed. Call markTaskFailed, which runs the failure 
listeners as part of
+          // this invocation: this thread stays registered as 
`listenerInvocationThread`, so that
+          // no other thread takes over the listeners in the meantime.
           //
           // One of the following cases applies (#3 being the interesting one):
           //
@@ -427,16 +647,11 @@ private[spark] class TaskContextImpl(
           //    [[markTaskFailed]] here will have no effect and we simply 
resume running the
           //    remaining completion handlers.
           try {
-            listenerInvocationThread = None
-            markTaskFailed(e)
+            markTaskFailed(e, Nested)
           } catch {
-            case t: Throwable => e.addSuppressed(t)
-          } finally {
-            synchronized {
-              if (listenerInvocationThread.isEmpty) {
-                listenerInvocationThread = Some(Thread.currentThread())
-              }
-            }
+            // Failures of the failure listeners. A fatal error (see 
`Utils.isFatalError()`, e.g.
+            // an OutOfMemoryError) propagates, through the release below.
+            case t if !Utils.isFatalError(t) => e.addSuppressed(t)
           }
           listenerExceptions += e
           // Best effort: a failure to log, e.g. from an appender that does 
not ignore exceptions,
@@ -449,11 +664,54 @@ private[spark] class TaskContextImpl(
           }
       }
     }
+
+    def runLeftFailureListeners(cause: Throwable): Unit = {
+      try {
+        invokeTaskFailureListeners(cause, Nested)
+      } catch {
+        // Merge the listeners' own failures, which the nested invocation has 
logged, not the
+        // exception that wraps them. A fatal error (see 
`Utils.isFatalError()`, e.g. an
+        // OutOfMemoryError) propagates, through the release below.
+        case e: TaskCompletionListenerException => listenerExceptions ++= 
e.getSuppressed.toSeq
+        case t if !Utils.isFatalError(t) => listenerExceptions += t
+      }
+    }
+
+    // Chooses the next thing to do under the TaskContext lock, to be done 
outside of it.
+    def nextStep(): Option[() => Unit] = synchronized {
+      if (completesTask && failureCauseOpt.isDefined && 
!onFailureCallbacks.empty()) {
+        val cause = failureCauseOpt.get
+        Some(() => runLeftFailureListeners(cause))
+      } else if (!listeners.empty()) {
+        val listener = listeners.pop()
+        Some(() => runListener(listener))
+      } else {
+        // We have executed all listeners that have been added so far. 
Deregister this thread as
+        // the callback invocation thread, or hand the invocation over to a 
waiting thread.
+        if (mode != Nested) {
+          releaseListenerInvocation()
+        }
+        None
+      }
+    }
+
+    try {
+      var step: Option[() => Unit] = None
+      while ({step = nextStep(); step.nonEmpty}) {
+        step.get()
+      }
+      released = true
+    } finally {
+      if (!released) {
+        // Something escaped the loop above. Do not leave this thread 
registered, which would
+        // keep any other thread from running listeners, or a waiting thread 
waiting forever.
+        synchronized {
+          releaseListenerInvocation()
+        }
+      }
+    }
     if (listenerExceptions.nonEmpty) {
-      val exception = new TaskCompletionListenerException(
-        listenerExceptions.map(_.getMessage).toSeq, error)
-      listenerExceptions.foreach(exception.addSuppressed)
-      throw exception
+      throw newTaskCompletionListenerException(listenerExceptions.toSeq, error)
     }
   }
 
@@ -557,3 +815,25 @@ private[spark] class TaskContextImpl(
     }
   }
 }
+
+private object TaskContextImpl {
+
+  /** How an `invokeListeners()` call takes part in the listener invocation. */
+  sealed trait InvocationMode
+
+  /**
+   * Registers this thread as `listenerInvocationThread` if no thread is 
registered; otherwise
+   * leaves the listeners to the registered thread, which may be this one, and 
which runs them only
+   * in some cases (see `invokeListeners()`).
+   */
+  case object Claim extends InvocationMode
+
+  /** The caller has already registered this thread as 
`listenerInvocationThread`. */
+  case object Claimed extends InvocationMode
+
+  /** An outer invocation on this thread has registered it, and keeps it when 
this one is done. */
+  case object Nested extends InvocationMode
+
+  /** How often to log while `markTaskCompleted()` waits for another thread 
invoking listeners. */
+  val DEFAULT_LISTENER_HANDOVER_WARN_INTERVAL_MS = 10000L
+}
diff --git a/core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala 
b/core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala
index 84109d042540..f0a918cf8637 100644
--- a/core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala
+++ b/core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala
@@ -378,7 +378,8 @@ private[spark] object BasePythonRunner extends Logging {
       // An interrupt of this thread that arrives during the wait (e.g. a task 
kill) does not stop
       // it and is not passed on (see stopAndAwaitExit): the kill is already 
recorded in the
       // TaskContext, and the flag would make the blocking channel and lock 
operations of the
-      // completion listeners that run next fail and skip cleanup.
+      // completion listeners that run next fail and skip cleanup (see also
+      // TaskContextImpl.awaitListenerInvocationHandover, which passes 
interrupts on).
       val intervalNs = TimeUnit.MILLISECONDS.toNanos(warnIntervalMs)
       val startNs = System.nanoTime()
       var nextWarningNs = startNs + intervalNs
diff --git 
a/core/src/main/scala/org/apache/spark/shuffle/streaming/StreamingShuffleReader.scala
 
b/core/src/main/scala/org/apache/spark/shuffle/streaming/StreamingShuffleReader.scala
index 70ea96154344..160a01794d32 100644
--- 
a/core/src/main/scala/org/apache/spark/shuffle/streaming/StreamingShuffleReader.scala
+++ 
b/core/src/main/scala/org/apache/spark/shuffle/streaming/StreamingShuffleReader.scala
@@ -400,8 +400,8 @@ class StreamingShuffleReader[K, C](
   private def checkTaskFailure(): Unit = {
     context.getTaskFailure.foreach(throw _)
     // Surface any background error on the task thread so that markTaskFailed 
and
-    // markTaskCompleted are called from this thread, ensuring completion 
listeners
-    // (including cleanupResources) run without contention.
+    // markTaskCompleted are called from this thread, so that the failure and 
completion
+    // listeners (including cleanupResources) run here without waiting for 
another thread.
     errorNotifier.throwErrorIfExists()
     if (context.isInterrupted()) {
       throw new InterruptedException("Task interrupted. Exiting read loop.")
diff --git a/core/src/main/scala/org/apache/spark/util/ErrorNotifier.scala 
b/core/src/main/scala/org/apache/spark/util/ErrorNotifier.scala
index a40e0c7bd524..5660ffb427c6 100644
--- a/core/src/main/scala/org/apache/spark/util/ErrorNotifier.scala
+++ b/core/src/main/scala/org/apache/spark/util/ErrorNotifier.scala
@@ -28,9 +28,10 @@ import org.apache.spark.internal.Logging
  *
  * A background thread records the first error via [[markError]] rather than 
failing the task
  * directly; the owning thread polls [[throwErrorIfExists]] (or [[getError]]) 
at a safe point and
- * re-throws on its own thread. This avoids a race where calling 
`TaskContext.markTaskFailed` from
- * a background thread can run concurrently with the task thread's 
`markTaskCompleted` and silently
- * skip completion listeners (including cleanup), leading to thread leaks.
+ * re-throws on its own thread. Calling `TaskContext.markTaskFailed` from a 
background thread
+ * can instead run the failure listeners on that thread, if no other thread is 
invoking listeners,
+ * and a concurrent `markTaskCompleted` on the task thread then waits for them 
before it runs the
+ * completion listeners (including cleanup).
  *
  * Shared by the streaming shuffle and the streaming-query async checkpointing 
paths.
  */
diff --git 
a/core/src/test/scala/org/apache/spark/scheduler/TaskContextSuite.scala 
b/core/src/test/scala/org/apache/spark/scheduler/TaskContextSuite.scala
index bc4e6cbe76ce..f1454ac03b5c 100644
--- a/core/src/test/scala/org/apache/spark/scheduler/TaskContextSuite.scala
+++ b/core/src/test/scala/org/apache/spark/scheduler/TaskContextSuite.scala
@@ -22,14 +22,17 @@ import java.util.concurrent.{ConcurrentLinkedQueue, 
CountDownLatch, Semaphore, T
 import java.util.concurrent.atomic.{AtomicInteger, AtomicReference}
 
 import scala.collection.mutable.ArrayBuffer
+import scala.concurrent.duration._
 import scala.jdk.CollectionConverters._
 
+import org.apache.logging.log4j.Level
 import org.apache.logging.log4j.core.LogEvent
 import org.apache.logging.log4j.core.appender.AbstractAppender
 import org.apache.logging.log4j.core.config.Property
 import org.mockito.ArgumentMatchers.{any, eq => mockitoEq}
 import org.mockito.Mockito._
 import org.scalatest.BeforeAndAfter
+import org.scalatest.concurrent.Eventually.{eventually, timeout}
 
 import org.apache.spark._
 import org.apache.spark.api.plugin.{DriverPlugin, ExecutorPlugin, SparkPlugin}
@@ -398,23 +401,22 @@ class TaskContextSuite extends SparkFunSuite with 
BeforeAndAfter with LocalSpark
     })
 
     // Interrupt the task from a separate thread and wait until the interrupt 
listener starts.
-    val interruptThread = new Thread(() => context.markInterrupted("test 
interrupt"))
-    interruptThread.start()
-    interruptListenerStarted.acquire()
-
-    // While the interrupt listener is running on the interrupt thread, mark 
the task completed
-    // on this thread. With the dedicated interrupt listener loop, this must 
NOT be blocked.
-    context.markTaskCompleted(None)
-
-    // The completion listener should have been called even though the 
interrupt listener is still
-    // running. If `invokeListeners()` were shared between interrupt and 
completion listeners,
-    // the completion listener would be silently skipped because 
`listenerInvocationThread` would
-    // be held by the interrupt thread.
-    verify(completionListener, times(1)).onTaskCompletion(any())
-
-    // Release the interrupt listener and join the interrupt thread.
-    interruptListenerRelease.release()
-    interruptThread.join()
+    val interruptThread = 
startDaemonThread("interrupt")(context.markInterrupted("test interrupt"))
+    try {
+      interruptListenerStarted.acquire()
+
+      // While the interrupt listener is running on the interrupt thread, mark 
the task completed
+      // on another thread. With the dedicated interrupt listener loop, this 
must NOT be blocked.
+      // If `invokeListeners()` were shared between interrupt and completion 
listeners, the
+      // interrupt thread would hold `listenerInvocationThread`, and the 
completion would wait
+      // for it, so the join below would time out.
+      
joinThread(startDaemonThread("completion")(context.markTaskCompleted(None)))
+      verify(completionListener, times(1)).onTaskCompletion(any())
+    } finally {
+      // Release the interrupt listener and join the interrupt thread.
+      interruptListenerRelease.release()
+    }
+    joinThread(interruptThread)
   }
 
   test("FailureListener throws after task body fails") {
@@ -858,14 +860,9 @@ class TaskContextSuite extends SparkFunSuite with 
BeforeAndAfter with LocalSpark
     // Create a listener that registers another listener using a second thread.
     val multithreadedListener = new TaskCompletionListener {
       override def onTaskCompletion(context: TaskContext): Unit = {
-        val thread = new Thread(new Runnable {
-          override def run(): Unit = {
-            context.addTaskCompletionListener(simpleListener)
-          }
-        })
-        thread.start()
+        val thread = 
startDaemonThread("adder")(context.addTaskCompletionListener(simpleListener))
         invocations.getAndIncrement()
-        thread.join()
+        joinThread(thread)
       }
     }
     context.addTaskCompletionListener(multithreadedListener)
@@ -1195,19 +1192,721 @@ class TaskContextSuite extends SparkFunSuite with 
BeforeAndAfter with LocalSpark
     completion
   }
 
+  /** A daemon thread that records what `body` throws, for `joinThread` to 
check. */
+  private class DaemonThread(name: String, body: => Unit) extends Thread(name) 
{
+    setDaemon(true)
+    @volatile var error: Throwable = _
+    override def run(): Unit = {
+      try {
+        body
+      } catch {
+        case t: Throwable => error = t
+      }
+    }
+  }
+
+  private def startDaemonThread(name: String)(body: => Unit): DaemonThread = {
+    val thread = new DaemonThread(name, body)
+    thread.start()
+    thread
+  }
+
+  private def joinThread(thread: DaemonThread): Unit = {
+    thread.join(TimeUnit.MINUTES.toMillis(1))
+    assert(!thread.isAlive, s"${thread.getName} is still running")
+    Option(thread.error).foreach(e => fail(s"${thread.getName} failed", e))
+  }
+
+  /**
+   * Runs `takeInvocation` on a new thread named `name`, by default failing 
the task, and waits
+   * until a failure listener on it holds the listener invocation, which it 
does until the
+   * returned latch is counted down. Returns that latch and the thread.
+   */
+  private def holdListenerInvocation(context: TaskContext, name: String = 
"listener-holder")(
+      takeInvocation: => Unit = context.markTaskFailed(new Exception("failed 
on another thread")))
+      : (CountDownLatch, DaemonThread) = {
+    val holding = new CountDownLatch(1)
+    val release = new CountDownLatch(1)
+    context.addTaskFailureListener(new TaskFailureListener {
+      override def onTaskFailure(context: TaskContext, error: Throwable): Unit 
= {
+        holding.countDown()
+        release.await(1, TimeUnit.MINUTES)
+      }
+    })
+    val holder = startDaemonThread(name)(takeInvocation)
+    assert(holding.await(1, TimeUnit.MINUTES))
+    (release, holder)
+  }
+
   /**
-   * An appender that throws `failure` for every event whose message contains 
`trigger`. Unlike
-   * `LogAppender`, it does not ignore exceptions, so log4j throws them from 
the logging call.
+   * Fails every logging call whose message contains `trigger` with `failure`. 
Unlike
+   * `LogAppender`, it does not ignore its exceptions, so log4j passes them to 
the caller.
    */
   private class FailingAppender(trigger: String, failure: RuntimeException)
     extends AbstractAppender("failingAppender", null, null, false, 
Property.EMPTY_ARRAY) {
+    val failures = new AtomicInteger()
+    // Of these, the failures of events that carry a throwable.
+    val failuresWithThrown = new AtomicInteger()
     override def append(event: LogEvent): Unit = {
       if (event.getMessage.getFormattedMessage.contains(trigger)) {
+        failures.incrementAndGet()
+        if (event.getThrown != null) failuresWithThrown.incrementAndGet()
         throw failure
       }
     }
   }
 
+  private def isCausedBy(error: Throwable, cause: Throwable): Boolean =
+    Iterator.iterate(error)(_.getCause).takeWhile(_ != null).exists(_ eq cause)
+
+  /** An exception whose `getMessage`, which is listener code too, throws 
`failure`. */
+  private class MessageFailingException(failure: Throwable) extends Exception {
+    override def getMessage: String = throw failure
+  }
+
+  private def addFailureListenerThrowing(context: TaskContext)(failure: => 
Throwable): Unit =
+    context.addTaskFailureListener(new TaskFailureListener {
+      override def onTaskFailure(context: TaskContext, error: Throwable): Unit 
= throw failure
+    })
+
+  /** Waits until `thread` waits in `markTaskCompleted()` for the listener 
invocation. */
+  private def awaitHandoverWait(thread: Thread): Unit = {
+    eventually(timeout(1.minute)) {
+      assert(!thread.isAlive || thread.getState == Thread.State.TIMED_WAITING)
+    }
+    assert(thread.isAlive, s"${thread.getName} returned instead of waiting")
+  }
+
+  test("SPARK-60222: a listener added as completion starts does not take over 
the listeners") {
+    // A thread that sees the task completed and adds a listener at that 
moment must not become
+    // the thread that runs the pending listeners: the completing thread 
claims the listener
+    // invocation together with setting `completed`.
+    (1 to 500).foreach { _ =>
+      val context = TaskContext.empty()
+      val pendingListenerThread = new AtomicReference[Thread]()
+      context.addTaskCompletionListener[Unit] { _ =>
+        pendingListenerThread.set(Thread.currentThread())
+      }
+      val ready = new CountDownLatch(1)
+      val adder = startDaemonThread("adder") {
+        ready.countDown()
+        while (!context.isCompleted()) {}
+        context.addTaskCompletionListener[Unit] { _ => }
+      }
+      ready.await()
+      context.markTaskCompleted(None)
+      joinThread(adder)
+      assert(pendingListenerThread.get eq Thread.currentThread())
+    }
+  }
+
+  test("SPARK-60222: a failing listener does not let another thread take over 
the listeners") {
+    // While a failing listener's failure listeners run, the thread invoking 
the listeners must
+    // stay registered, so that another thread adding listeners at that moment 
does not start
+    // running the pending ones, possibly concurrently with this thread.
+    (1 to 200).foreach { _ =>
+      val context = TaskContext.empty()
+      val pendingListenerThread = new AtomicReference[Thread]()
+      context.addTaskCompletionListener[Unit] { _ =>
+        pendingListenerThread.set(Thread.currentThread())
+      }
+      // Two failing listeners, which run first: the second failure finds the 
task already failed.
+      (1 to 2).foreach { i =>
+        context.addTaskCompletionListener(new TaskCompletionListener {
+          override def onTaskCompletion(context: TaskContext): Unit = throw 
new Exception(s"$i")
+        })
+      }
+      context.addTaskFailureListener(new TaskFailureListener {
+        override def onTaskFailure(context: TaskContext, error: Throwable): 
Unit = {}
+      })
+      val ready = new CountDownLatch(1)
+      val adder = startDaemonThread("adder") {
+        ready.countDown()
+        while (!context.isCompleted()) {}
+        (1 to 200).foreach { _ => context.addTaskCompletionListener[Unit] { _ 
=> } }
+      }
+      ready.await()
+      intercept[TaskCompletionListenerException] {
+        context.markTaskCompleted(None)
+      }
+      joinThread(adder)
+      assert(pendingListenerThread.get eq Thread.currentThread())
+    }
+  }
+
+  test("SPARK-60222: completion waits for another thread invoking listeners 
and runs them") {
+    val context = TaskContext.empty()
+    val listenerThreads = new ConcurrentLinkedQueue[(String, Thread)]()
+    context.addTaskCompletionListener(loggingListener(listenerThreads, 
"pending"))
+    context.addTaskInterruptListener(new TaskInterruptListener {
+      override def onTaskInterrupted(context: TaskContext, reason: String): 
Unit =
+        throw new Exception("interrupt listener")
+    })
+    // Like a kill: an interrupt listener fails, so the kill thread runs the 
failure listeners.
+    val (release, killer) = holdListenerInvocation(context, "killer") {
+      context.markInterrupted("killed")
+    }
+
+    val completion = 
startDaemonThread("completion")(context.markTaskCompleted(None))
+    try {
+      awaitHandoverWait(completion)
+      // Neither returns early nor runs the completion listeners while the 
kill thread holds the
+      // listener invocation, and a third thread adding a listener does not 
take them over.
+      joinThread(startDaemonThread("adder") {
+        context.addTaskCompletionListener(loggingListener(listenerThreads, 
"added"))
+      })
+      assert(completion.isAlive)
+      assert(listenerThreads.isEmpty)
+    } finally {
+      release.countDown()
+    }
+    joinThread(completion)
+    joinThread(killer)
+    assert(listenerThreads.asScala.toSeq === Seq(("added", completion), 
("pending", completion)))
+  }
+
+  test("SPARK-60222: a listener on the invoking thread completing the task 
runs them nested") {
+    val context = TaskContext.empty()
+    val completionListenerThread = new AtomicReference[Thread]()
+    context.addTaskCompletionListener[Unit] { _ =>
+      completionListenerThread.set(Thread.currentThread())
+    }
+    val ranBeforeReturn = new AtomicReference[Option[Boolean]](None)
+    val laterListenerThread = new AtomicReference[Thread]()
+    context.addTaskFailureListener(new TaskFailureListener {
+      override def onTaskFailure(context: TaskContext, error: Throwable): Unit 
= {
+        // This thread holds the listener invocation, so it must not wait for 
itself.
+        context.markTaskCompleted(Some(error))
+        ranBeforeReturn.set(Some(completionListenerThread.get eq 
Thread.currentThread()))
+        // And it still holds the invocation, so another thread adding a 
listener does not run it.
+        joinThread(startDaemonThread("adder") {
+          context.addTaskCompletionListener[Unit] { _ =>
+            laterListenerThread.set(Thread.currentThread())
+          }
+        })
+      }
+    })
+    joinThread(startDaemonThread("failer")(context.markTaskFailed(new 
Exception("failed"))))
+    assert(ranBeforeReturn.get === Some(true))
+    assert(laterListenerThread.get == null)
+  }
+
+  test("SPARK-60222: a failure listener left to the completing thread runs 
before the next " +
+      "completion listener") {
+    val context = TaskContext.empty()
+    val invocations = new ConcurrentLinkedQueue[(String, Thread)]()
+    context.addTaskFailureListener(new TaskFailureListener {
+      override def onTaskFailure(context: TaskContext, error: Throwable): Unit 
= {
+        invocations.add(("failure", Thread.currentThread()))
+        throw new Exception("failure listener")
+      }
+    })
+    // E.g. a completion listener that releases a state store, which a failure 
listener aborts.
+    context.addTaskCompletionListener(loggingListener(invocations, "second"))
+    context.addTaskCompletionListener[Unit] { _ =>
+      invocations.add(("first", Thread.currentThread()))
+      // Another thread fails the task while this thread runs the completion 
listeners, e.g. a
+      // kill whose interrupt listener failed. It finds the listener 
invocation taken and leaves
+      // the failure listeners to this thread.
+      joinThread(startDaemonThread("failer") {
+        context.markTaskFailed(new Exception("failed elsewhere"))
+      })
+      assert(invocations.size == 1)
+    }
+
+    val error = intercept[TaskCompletionListenerException] {
+      context.markTaskCompleted(None)
+    }
+    val current = Thread.currentThread()
+    assert(invocations.asScala.toSeq ===
+      Seq(("first", current), ("failure", current), ("second", current)))
+    // The failure listener's failure is reported as its own, not wrapped in 
the exception of the
+    // failure listener invocation.
+    assert(error.getSuppressed.map(_.getMessage).toSeq === Seq("failure 
listener"))
+  }
+
+  test("SPARK-60222: a failure listener left to the thread the invocation was 
handed over to " +
+      "runs before its next completion listener") {
+    val context = TaskContext.empty()
+    val invocations = new ConcurrentLinkedQueue[(String, Thread)]()
+    context.addTaskCompletionListener(loggingListener(invocations, "second"))
+    context.addTaskCompletionListener[Unit] { _ =>
+      invocations.add(("first", Thread.currentThread()))
+      // The task has failed, so the added failure listener is to run right 
away. This thread
+      // holds the listener invocation, so the adding thread leaves it to this 
one.
+      joinThread(startDaemonThread("adder") {
+        context.addTaskFailureListener(new TaskFailureListener {
+          override def onTaskFailure(context: TaskContext, error: Throwable): 
Unit =
+            invocations.add(("failure", Thread.currentThread()))
+        })
+      })
+      assert(invocations.size == 1)
+    }
+    val (release, holder) = holdListenerInvocation(context)()
+    val completion = 
startDaemonThread("completion")(context.markTaskCompleted(None))
+    try {
+      awaitHandoverWait(completion)
+    } finally {
+      release.countDown()
+    }
+    joinThread(completion)
+    joinThread(holder)
+    assert(invocations.asScala.toSeq ===
+      Seq(("first", completion), ("failure", completion), ("second", 
completion)))
+  }
+
+  test("SPARK-60222: a failure of getMessage is reported in place of the 
exception it fails for") {
+    val messageFailure = new IllegalStateException("no message")
+    val current = Thread.currentThread()
+    // The failure listener invocation that a failing pre-completion or 
completion listener
+    // starts reports the failure of `getMessage`, and the listener loops go 
on.
+    Seq(true, false).foreach { preCompletion =>
+      val context = TaskContext.empty()
+      val invocations = new ConcurrentLinkedQueue[(String, Thread)]()
+      addFailureListenerThrowing(context)(new 
MessageFailingException(messageFailure))
+      context.addTaskCompletionListener(loggingListener(invocations, "next"))
+      val failing = new TaskCompletionListener {
+        override def onTaskCompletion(context: TaskContext): Unit = throw new 
Exception("listener")
+      }
+      if (preCompletion) {
+        context.addTaskPreCompletionListener(failing)
+      } else {
+        context.addTaskCompletionListener(failing)
+      }
+      val error = intercept[TaskCompletionListenerException] {
+        context.markTaskCompleted(None)
+      }
+      assert(invocations.asScala.toSeq === Seq(("next", current)), 
s"preCompletion=$preCompletion")
+      assert(context.isCompleted())
+      val listenerFailure = error.getSuppressed.toSeq.head
+      assert(listenerFailure.getMessage === "listener")
+      assert(listenerFailure.getSuppressed.toSeq.flatMap(_.getSuppressed) === 
Seq(messageFailure))
+    }
+
+    // The failures of the other listeners are kept: of the failure listeners 
left to the
+    // completing thread,
+    {
+      val context = TaskContext.empty()
+      val invocations = new ConcurrentLinkedQueue[(String, Thread)]()
+      val otherFailure = new Exception("a")
+      addFailureListenerThrowing(context)(otherFailure)
+      addFailureListenerThrowing(context)(new 
MessageFailingException(messageFailure))
+      context.addTaskCompletionListener(loggingListener(invocations, "second"))
+      context.addTaskCompletionListener[Unit] { _ =>
+        joinThread(startDaemonThread("failer") {
+          context.markTaskFailed(new Exception("failed elsewhere"))
+        })
+      }
+      val error = intercept[TaskCompletionListenerException] {
+        context.markTaskCompleted(None)
+      }
+      assert(invocations.asScala.toSeq === Seq(("second", current)))
+      assert(error.getSuppressed.toSeq === Seq(messageFailure, otherFailure))
+      assert(error.getMessage.contains("no message") && 
error.getMessage.contains("a"))
+    }
+
+    // of the pre-completion listeners,
+    {
+      val context = TaskContext.empty()
+      val otherFailure = new Exception("a")
+      Seq(otherFailure, new MessageFailingException(messageFailure)).foreach { 
failure =>
+        context.addTaskPreCompletionListener(new TaskCompletionListener {
+          override def onTaskCompletion(context: TaskContext): Unit = throw 
failure
+        })
+      }
+      val error = intercept[TaskCompletionListenerException] {
+        context.markTaskCompleted(None)
+      }
+      assert(context.isCompleted())
+      assert(error.getSuppressed.toSeq === Seq(messageFailure, otherFailure))
+    }
+
+    // and of the interrupt listeners, whose failure is recorded instead of 
escaping the kill.
+    {
+      val context = TaskContext.empty()
+      val otherFailure = new Exception("a")
+      Seq(otherFailure, new MessageFailingException(messageFailure)).foreach { 
failure =>
+        context.addTaskInterruptListener(new TaskInterruptListener {
+          override def onTaskInterrupted(context: TaskContext, reason: 
String): Unit =
+            throw failure
+        })
+      }
+      context.markInterrupted("killed")
+      val recorded = context.getTaskFailure
+      assert(recorded.exists(_.isInstanceOf[TaskCompletionListenerException]), 
recorded)
+      assert(recorded.get.getSuppressed.toSeq.take(2) === Seq(messageFailure, 
otherFailure))
+    }
+  }
+
+  test("SPARK-60222: an error that is not fatal to the executor does not end 
the listener loops") {
+    // A `LinkageError` is not `NonFatal`, but the executor keeps running. It 
escapes the failure
+    // listener invocation if the failure of a `getMessage` fails in 
`getMessage` too.
+    val linkageError = new LinkageError("no message")
+    def failure: Throwable =
+      new MessageFailingException(new MessageFailingException(linkageError))
+    val current = Thread.currentThread()
+    // After a failing pre-completion or completion listener,
+    Seq(true, false).foreach { preCompletion =>
+      val context = TaskContext.empty()
+      val invocations = new ConcurrentLinkedQueue[(String, Thread)]()
+      addFailureListenerThrowing(context)(failure)
+      context.addTaskCompletionListener(loggingListener(invocations, "next"))
+      val failing = new TaskCompletionListener {
+        override def onTaskCompletion(context: TaskContext): Unit = throw new 
Exception("listener")
+      }
+      if (preCompletion) {
+        context.addTaskPreCompletionListener(failing)
+      } else {
+        context.addTaskCompletionListener(failing)
+      }
+      val error = intercept[TaskCompletionListenerException] {
+        context.markTaskCompleted(None)
+      }
+      assert(invocations.asScala.toSeq === Seq(("next", current)), 
s"preCompletion=$preCompletion")
+      assert(context.isCompleted())
+      val listenerFailure = error.getSuppressed.toSeq.head
+      assert(listenerFailure.getMessage === "listener")
+      assert(listenerFailure.getSuppressed.exists(_ eq linkageError))
+    }
+
+    // For the failure listeners left to the completing thread,
+    {
+      val context = TaskContext.empty()
+      val invocations = new ConcurrentLinkedQueue[(String, Thread)]()
+      addFailureListenerThrowing(context)(failure)
+      context.addTaskCompletionListener(loggingListener(invocations, "second"))
+      context.addTaskCompletionListener[Unit] { _ =>
+        joinThread(startDaemonThread("failer") {
+          context.markTaskFailed(new Exception("failed elsewhere"))
+        })
+      }
+      val error = intercept[TaskCompletionListenerException] {
+        context.markTaskCompleted(None)
+      }
+      assert(invocations.asScala.toSeq === Seq(("second", current)))
+      assert(error.getSuppressed.toSeq === Seq(linkageError))
+    }
+
+    // and when the completion listener invocation throws it after a 
pre-completion listener
+    // failed, it is reported with that failure.
+    val context = TaskContext.empty()
+    val preCompletionFailure = new Exception("pre-completion listener")
+    context.addTaskPreCompletionListener(new TaskCompletionListener {
+      override def onTaskCompletion(context: TaskContext): Unit = throw 
preCompletionFailure
+    })
+    context.addTaskCompletionListener[Unit] { _ => throw failure }
+    val error = intercept[TaskCompletionListenerException] {
+      context.markTaskCompleted(None)
+    }
+    assert(error.getSuppressed.toSeq === Seq(preCompletionFailure, 
linkageError))
+  }
+
+  test("SPARK-60222: an error that escapes the listener loop still releases 
the invocation") {
+    val context = TaskContext.empty()
+    context.markTaskCompleted(None)
+    val fatal = new OutOfMemoryError("no message")
+    addFailureListenerThrowing(context)(new MessageFailingException(fatal))
+    val error = intercept[OutOfMemoryError] {
+      // Runs right away on this thread and fails, and the failure listener 
invocation that this
+      // starts throws `fatal` while building its exception.
+      context.addTaskCompletionListener[Unit] { _ => throw new 
Exception("listener") }
+    }
+    assert(error eq fatal)
+    // Released: a listener added now runs right away on this thread.
+    val laterListenerThread = new AtomicReference[Thread]()
+    context.addTaskCompletionListener[Unit] { _ =>
+      laterListenerThread.set(Thread.currentThread())
+    }
+    assert(laterListenerThread.get eq Thread.currentThread())
+  }
+
+  test("SPARK-60222: a fatal error of a failure listener invocation is not 
hidden by the failure " +
+      "of the listener that started it") {
+    val fatal = new OutOfMemoryError("no message")
+    // Both for a failing completion listener and for a failing pre-completion 
listener.
+    Seq(true, false).foreach { preCompletion =>
+      val context = TaskContext.empty()
+      addFailureListenerThrowing(context)(new MessageFailingException(fatal))
+      val failing = new TaskCompletionListener {
+        override def onTaskCompletion(context: TaskContext): Unit = throw new 
Exception("listener")
+      }
+      if (preCompletion) {
+        context.addTaskPreCompletionListener(failing)
+      } else {
+        context.addTaskCompletionListener(failing)
+      }
+      // It propagates, e.g. so that the executor sees it if the task body 
succeeded, instead of
+      // being a suppressed exception of the listener's failure.
+      val error = intercept[OutOfMemoryError] {
+        context.markTaskCompleted(None)
+      }
+      assert(error eq fatal, s"preCompletion=$preCompletion")
+    }
+  }
+
+  test("SPARK-60222: an interrupt does not stop the handover wait and is 
kept") {
+    val context = TaskContext.empty()
+    val listenerThreads = new ConcurrentLinkedQueue[(String, Thread)]()
+    (1 to 2).foreach { i =>
+      context.addTaskCompletionListener(loggingListener(listenerThreads, 
s"completion $i"))
+    }
+    val (release, holder) = holdListenerInvocation(context)()
+    val interruptedAfterCompletion = new AtomicReference[Option[Boolean]](None)
+    val completion = startDaemonThread("completion") {
+      context.markTaskCompleted(None)
+      
interruptedAfterCompletion.set(Some(Thread.currentThread().isInterrupted))
+    }
+    try {
+      awaitHandoverWait(completion)
+      completion.interrupt()
+      // Still waiting for the holder, without having run any completion 
listener.
+      eventually(timeout(1.minute)) {
+        assert(!completion.isAlive ||
+          !completion.isInterrupted && completion.getState == 
Thread.State.TIMED_WAITING)
+      }
+      assert(completion.isAlive, "completion returned instead of waiting")
+      assert(listenerThreads.isEmpty)
+    } finally {
+      release.countDown()
+    }
+    joinThread(completion)
+    joinThread(holder)
+    assert(interruptedAfterCompletion.get === Some(true))
+    assert(listenerThreads.asScala.toSeq ===
+      Seq(("completion 2", completion), ("completion 1", completion)))
+  }
+
+  test("SPARK-60222: the handover wait warns with the holder's name and stack 
trace") {
+    val context = TaskContext.empty()
+    context.listenerHandoverWarnIntervalMs = 100
+    val (release, holder) = holdListenerInvocation(context)()
+    val logAppender = new LogAppender("listener handover wait")
+    var completion: DaemonThread = null
+    try {
+      withLogAppender(logAppender, Seq(classOf[TaskContextImpl].getName), 
Some(Level.WARN)) {
+        completion = 
startDaemonThread("completion")(context.markTaskCompleted(None))
+        eventually(timeout(1.minute)) {
+          assert(!completion.isAlive ||
+            logAppender.loggingEvents.count(_.getLevel == Level.WARN) >= 2)
+        }
+        assert(completion.isAlive, "completion returned instead of waiting")
+      }
+    } finally {
+      release.countDown()
+    }
+    joinThread(completion)
+    joinThread(holder)
+    val warnings = logAppender.loggingEvents.filter(_.getLevel == 
Level.WARN).toSeq
+    warnings.foreach { event =>
+      assert(event.getMessage.getFormattedMessage.contains("until thread 
listener-holder"))
+    }
+    // The first warning carries the holder's stack trace, blocked in its 
failure listener.
+    val holderStack = warnings.head.getThrown
+    assert(holderStack.getMessage.contains("listener-holder"))
+    assert(holderStack.getStackTrace.exists(_.getMethodName == 
"onTaskFailure"))
+    // The later ones do not walk the stack again.
+    warnings.tail.foreach { event => assert(event.getThrown == null) }
+  }
+
+  test("SPARK-60222: a failing handover warning does not end the wait") {
+    val context = TaskContext.empty()
+    context.listenerHandoverWarnIntervalMs = 50
+    val listenerThreads = new ConcurrentLinkedQueue[(String, Thread)]()
+    context.addTaskCompletionListener(loggingListener(listenerThreads, 
"pending"))
+    val (release, holder) = holdListenerInvocation(context)()
+    val logFailure = new IllegalStateException("cannot log")
+    val appender = new FailingAppender("has been waiting", logFailure)
+    val completionError = new AtomicReference[Throwable]()
+    var completion: DaemonThread = null
+    try {
+      withLogAppender(appender, Seq(classOf[TaskContextImpl].getName), 
Some(Level.WARN)) {
+        completion = startDaemonThread("completion") {
+          try {
+            context.markTaskCompleted(None)
+          } catch {
+            case t: Throwable => completionError.set(t)
+          }
+        }
+        eventually(timeout(1.minute)) {
+          assert(!completion.isAlive || appender.failures.get >= 2)
+        }
+        assert(completion.isAlive, "completion returned instead of waiting")
+        assert(listenerThreads.isEmpty)
+      }
+    } finally {
+      release.countDown()
+    }
+    joinThread(completion)
+    joinThread(holder)
+    // The completing thread runs the listeners once the holder has handed 
over, and then throws
+    // the first failure.
+    assert(listenerThreads.asScala.toSeq === Seq(("pending", completion)))
+    assert(isCausedBy(completionError.get, logFailure))
+    // Only the first attempt to warn took the holder's stack trace, although 
it failed.
+    assert(appender.failuresWithThrown.get === 1)
+    // And it has released the invocation: a listener added now runs right 
away on this thread.
+    val laterListenerThread = new AtomicReference[Thread]()
+    context.addTaskCompletionListener[Unit] { _ =>
+      laterListenerThread.set(Thread.currentThread())
+    }
+    assert(laterListenerThread.get eq Thread.currentThread())
+  }
+
+  test("SPARK-60222: a failing handover warning is reported with the failures 
of the " +
+      "completion listeners, unless one of them is fatal") {
+    // `fatal` is whether the completion listener invocation throws an 
OutOfMemoryError, while
+    // building its exception.
+    Seq(false, true).foreach { fatal =>
+      val context = TaskContext.empty()
+      context.listenerHandoverWarnIntervalMs = 50
+      val oom = new OutOfMemoryError("no message")
+      val listenerFailure =
+        if (fatal) new MessageFailingException(oom) else new 
Exception("completion listener")
+      context.addTaskCompletionListener[Unit] { _ => throw listenerFailure }
+      val (release, holder) = holdListenerInvocation(context)()
+      val logFailure = new IllegalStateException("cannot log")
+      val appender = new FailingAppender("has been waiting", logFailure)
+      val completionError = new AtomicReference[Throwable]()
+      var completion: DaemonThread = null
+      try {
+        withLogAppender(appender, Seq(classOf[TaskContextImpl].getName), 
Some(Level.WARN)) {
+          completion = startDaemonThread("completion") {
+            try {
+              context.markTaskCompleted(None)
+            } catch {
+              case t: Throwable => completionError.set(t)
+            }
+          }
+          eventually(timeout(1.minute)) {
+            assert(!completion.isAlive || appender.failures.get >= 1)
+          }
+        }
+      } finally {
+        release.countDown()
+      }
+      joinThread(completion)
+      joinThread(holder)
+      val error = completionError.get
+      if (fatal) {
+        // It propagates instead of becoming a suppressed exception of the 
failure to warn.
+        assert(error eq oom)
+      } else {
+        // The failure to warn, with the completion listeners' failure.
+        assert(isCausedBy(error, logFailure), error)
+        val reported = error.getSuppressed.toSeq
+        assert(reported.exists {
+          case e: TaskCompletionListenerException => e.getSuppressed.exists(_ 
eq listenerFailure)
+          case _ => false
+        }, reported)
+      }
+    }
+  }
+
+  test("SPARK-60222: a failing handover warning is reported with the failures 
of the " +
+      "pre-completion listeners") {
+    val context = TaskContext.empty()
+    context.listenerHandoverWarnIntervalMs = 50
+    val preCompletionFailure = new Exception("pre-completion listener")
+    context.addTaskPreCompletionListener(new TaskCompletionListener {
+      override def onTaskCompletion(context: TaskContext): Unit = throw 
preCompletionFailure
+    })
+    val (release, holder) = holdListenerInvocation(context)()
+    val logFailure = new IllegalStateException("cannot log")
+    val appender = new FailingAppender("has been waiting", logFailure)
+    val completionError = new AtomicReference[Throwable]()
+    var completion: DaemonThread = null
+    try {
+      withLogAppender(appender, Seq(classOf[TaskContextImpl].getName), 
Some(Level.WARN)) {
+        completion = startDaemonThread("completion") {
+          try {
+            context.markTaskCompleted(None)
+          } catch {
+            case t: Throwable => completionError.set(t)
+          }
+        }
+        eventually(timeout(1.minute)) {
+          assert(!completion.isAlive || appender.failures.get >= 1)
+        }
+      }
+    } finally {
+      release.countDown()
+    }
+    joinThread(completion)
+    joinThread(holder)
+    val error = completionError.get
+    assert(error.isInstanceOf[TaskCompletionListenerException], error)
+    val reported = error.getSuppressed.toSeq
+    // The failure of the pre-completion listener, which came first, then the 
failure to warn.
+    assert(reported.head eq preCompletionFailure)
+    assert(reported.tail.exists(isCausedBy(_, logFailure)))
+  }
+
+  test("SPARK-60222: the handover wait does not warn about a holder that has 
handed over") {
+    val context = TaskContext.empty()
+    context.listenerHandoverWarnIntervalMs = 50
+    val holding = new CountDownLatch(1)
+    val release = new CountDownLatch(1)
+    context.addTaskFailureListener(new TaskFailureListener {
+      override def onTaskFailure(context: TaskContext, error: Throwable): Unit 
= {
+        holding.countDown()
+        release.await(1, TimeUnit.MINUTES)
+      }
+    })
+    // While the first warning takes its stack trace, the holder hands over 
and goes on: here it
+    // even exits, so its stack trace no longer shows the listeners.
+    val holder = new DaemonThread("holder", context.markTaskFailed(new 
Exception("failed"))) {
+      override def getStackTrace: Array[StackTraceElement] = {
+        release.countDown()
+        join(TimeUnit.MINUTES.toMillis(1))
+        super.getStackTrace
+      }
+    }
+    holder.start()
+    assert(holding.await(1, TimeUnit.MINUTES))
+    val logAppender = new LogAppender("listener handover wait")
+    withLogAppender(logAppender, Seq(classOf[TaskContextImpl].getName), 
Some(Level.WARN)) {
+      
joinThread(startDaemonThread("completion")(context.markTaskCompleted(None)))
+    }
+    joinThread(holder)
+    assert(!logAppender.loggingEvents.exists(_.getLevel == Level.WARN))
+  }
+
+  test("SPARK-60222: listeners added after completion are not run by a thread 
invoking other " +
+      "listeners") {
+    val context = TaskContext.empty()
+    val error = new Exception("failed")
+    context.markTaskFailed(error)
+    context.markTaskCompleted(Some(error))
+    val holding = new CountDownLatch(1)
+    val release = new CountDownLatch(1)
+    context.addPostStatusUpdateListener(new PostStatusUpdateListener {
+      override def onStatusUpdateSent(context: TaskContext): Unit = {
+        holding.countDown()
+        release.await(1, TimeUnit.MINUTES)
+      }
+    })
+    // Like the task thread once the executor has released the task's block 
locks and memory.
+    val invoker = 
startDaemonThread("status-update")(context.invokePostStatusUpdateListeners())
+    assert(holding.await(1, TimeUnit.MINUTES))
+    val listenerThreads = new ConcurrentLinkedQueue[(String, Thread)]()
+    try {
+      // Find the invocation taken, so they do not run the new listeners.
+      context.addTaskCompletionListener(loggingListener(listenerThreads, 
"added"))
+      context.addTaskFailureListener(new TaskFailureListener {
+        override def onTaskFailure(context: TaskContext, error: Throwable): 
Unit =
+          listenerThreads.add(("failure", Thread.currentThread()))
+      })
+    } finally {
+      release.countDown()
+    }
+    joinThread(invoker)
+    assert(listenerThreads.isEmpty)
+    // A listener added once no thread is invoking listeners runs it, on the 
adding thread.
+    context.addTaskCompletionListener(loggingListener(listenerThreads, 
"later"))
+    val current = Thread.currentThread()
+    assert(listenerThreads.asScala.toSeq === Seq(("later", current), ("added", 
current)))
+  }
+
   test("SPARK-46480: Add isFailed in TaskContext") {
     val context = TaskContext.empty()
     var isFailed = false
diff --git 
a/core/src/test/scala/org/apache/spark/shuffle/streaming/StreamingShuffleSuite.scala
 
b/core/src/test/scala/org/apache/spark/shuffle/streaming/StreamingShuffleSuite.scala
index 578af94baa68..95571466e5ab 100644
--- 
a/core/src/test/scala/org/apache/spark/shuffle/streaming/StreamingShuffleSuite.scala
+++ 
b/core/src/test/scala/org/apache/spark/shuffle/streaming/StreamingShuffleSuite.scala
@@ -1372,68 +1372,91 @@ class StreamingShuffleSuite
     }
   }
 
-  // Demonstrates why background threads must not call 
context.markTaskFailed() directly:
-  // doing so can cause the task thread's markTaskCompleted() to silently skip 
completion
-  // listeners. In production, cleanupResources() is a completion listener, so 
skipping it
-  // leaks threads. The reader avoids this by routing background errors 
through a
-  // CompletableFuture so markTaskFailed/markTaskCompleted are both called 
from the task thread.
+  // Shows why background threads should not call context.markTaskFailed() 
directly: the
+  // failure listeners then run on that thread, and the task thread's 
markTaskCompleted() blocks
+  // until they finish before it runs the completion listeners. In production, 
cleanupResources()
+  // is a completion listener, so it waits for whatever the background 
thread's failure listeners
+  // wait for. The reader avoids this by routing background errors through an 
ErrorNotifier so
+  // that markTaskFailed/markTaskCompleted are both called from the task 
thread.
   test("background markTaskFailed races with task thread markTaskCompleted") {
     val readerContext = createTaskContext(sparkConf, partitionId = 0)
-    TaskContext.setTaskContext(readerContext)
 
-    // Latches to force the exact interleaving that triggers the race.
+    // Latches to force the interleaving.
     val failureListenerEntered = new CountDownLatch(1)
     val failureListenerProceed = new CountDownLatch(1)
 
-    @volatile var completionListenerRan = false
+    @volatile var completionListenerThread: Thread = null
 
-    // Add a failure listener that blocks, holding the 
listenerInvocationThread guard.
+    // Add a failure listener that blocks, holding the listener invocation.
     readerContext.addTaskFailureListener(new 
org.apache.spark.util.TaskFailureListener {
       override def onTaskFailure(context: TaskContext, error: Throwable): Unit 
= {
         failureListenerEntered.countDown()
-        failureListenerProceed.await()
+        failureListenerProceed.await(1, TimeUnit.MINUTES)
       }
     })
 
     // Add a completion listener -- stands in for cleanupResources() in 
production.
     readerContext.addTaskCompletionListener[Unit] { _ =>
-      completionListenerRan = true
+      completionListenerThread = Thread.currentThread()
     }
 
     val error = new RuntimeException("background discovery failure")
 
-    // Background thread calls markTaskFailed -> enters invokeListeners -> 
acquires
-    // listenerInvocationThread guard -> blocks in the failure listener above.
+    // Background thread calls markTaskFailed -> runs the failure listeners -> 
holds the listener
+    // invocation while it blocks in the failure listener above.
+    @volatile var bgThreadError: Throwable = null
     val bgThread = new Thread(() => {
       TaskContext.setTaskContext(readerContext)
-      readerContext.markTaskFailed(error)
+      try {
+        readerContext.markTaskFailed(error)
+      } catch {
+        case t: Throwable => bgThreadError = t
+      }
     })
+    bgThread.setDaemon(true)
     bgThread.start()
 
-    // Wait for the failure listener to start (background thread now holds the 
guard).
-    failureListenerEntered.await()
+    // Wait for the failure listener to start (background thread now holds the 
invocation).
+    assert(failureListenerEntered.await(1, TimeUnit.MINUTES))
 
-    // Task thread calls markTaskCompleted while the guard is held.
-    // invokeListeners sees listenerInvocationThread is set -> returns 
immediately,
-    // permanently skipping the completion listener.
-    readerContext.markTaskCompleted(Some(error))
-
-    // Release the failure listener so bgThread can finish.
-    failureListenerProceed.countDown()
-    bgThread.join()
-
-    // This documents the race: the completion listener was skipped. The 
reader avoids it by
-    // never calling markTaskFailed from a background thread.
-    assert(!completionListenerRan,
-      "Expected completion listener to be skipped due to the race condition. " 
+
-      "If this starts passing, TaskContextImpl.invokeListeners may have been 
fixed " +
-      "and this test can be updated.")
-
-    TaskContext.unset()
+    // Stands in for the task thread: it calls markTaskCompleted while the 
invocation is held,
+    // and waits until the background thread hands the invocation over to it.
+    @volatile var taskThreadError: Throwable = null
+    val taskThread = new Thread(() => {
+      TaskContext.setTaskContext(readerContext)
+      try {
+        readerContext.markTaskCompleted(Some(error))
+      } catch {
+        case t: Throwable => taskThreadError = t
+      }
+    })
+    taskThread.setDaemon(true)
+    taskThread.start()
+    try {
+      eventually(Timeout(1.minute)) {
+        assert(!taskThread.isAlive || taskThread.getState == 
Thread.State.TIMED_WAITING)
+      }
+      assert(taskThread.isAlive, "markTaskCompleted returned instead of 
waiting")
+      // The completion listener does not run while the failure listener 
blocks.
+      assert(completionListenerThread == null)
+    } finally {
+      // Release the failure listener, so that the background thread hands the 
invocation over.
+      failureListenerProceed.countDown()
+    }
+    taskThread.join(TimeUnit.MINUTES.toMillis(1))
+    assert(!taskThread.isAlive)
+    bgThread.join(TimeUnit.MINUTES.toMillis(1))
+    assert(!bgThread.isAlive)
+
+    // The completion listener ran on the thread that called markTaskCompleted.
+    assert(taskThreadError == null)
+    assert(bgThreadError == null)
+    assert(completionListenerThread eq taskThread)
   }
 
-  // With the CompletableFuture approach, the reader's cleanupResources runs 
because
-  // markTaskFailed is called from the task thread (no race with the 
completion listener).
+  // With the ErrorNotifier, the reader's cleanupResources runs on the task 
thread because
+  // markTaskFailed is called from the task thread too, so task completion 
does not wait for a
+  // background thread running the failure listeners.
   test("reader resources are cleaned up when client creation fails") {
     class FailingClientReader[K, C](
         handle: ShuffleHandle,
@@ -1472,8 +1495,8 @@ class StreamingShuffleSuite
 
       // Simulate the task framework (runTaskWithListeners): markTaskFailed 
then
       // markTaskCompleted, both on the task thread. With the fix, the 
background
-      // thread does NOT call markTaskFailed, so there's no contention on the
-      // listenerInvocationThread guard, and completion listeners run normally.
+      // thread does NOT call markTaskFailed, so no other thread holds the 
listener
+      // invocation, and the completion listeners run without waiting.
       readerContext.markTaskFailed(error)
       readerContext.markTaskCompleted(Some(error))
 
@@ -1485,9 +1508,9 @@ class StreamingShuffleSuite
   // On the writer side: when a background Netty thread records an error via
   // errorNotifier.markError (instead of calling context.markTaskFailed 
directly), the
   // writer's write() method surfaces the error on the task thread. Because 
markTaskFailed
-  // and markTaskCompleted are both called from the task thread, there is no 
contention on
-  // the listenerInvocationThread guard in TaskContextImpl, and the completion 
listener
-  // (cleanupResources) runs normally.
+  // and markTaskCompleted are both called from the task thread, no other 
thread holds the
+  // listener invocation in TaskContextImpl, and the completion listener 
(cleanupResources)
+  // runs without waiting.
   test("writer resources are cleaned up when server handler " +
     "receive fails via errorNotifier") {
     withSpark(new SparkContext("local", "StreamingShuffleSuite", sparkConf)) { 
sc =>
@@ -1546,9 +1569,9 @@ class StreamingShuffleSuite
 
       // Simulate the task framework (runTaskWithListeners): markTaskFailed 
then
       // markTaskCompleted, both on the task thread.  With the ErrorNotifier 
fix, the
-      // background Netty thread does NOT call markTaskFailed, so there is no 
contention
-      // on the listenerInvocationThread guard, and the completion listener
-      // (cleanupResources) runs normally.
+      // background Netty thread does NOT call markTaskFailed, so no other 
thread holds
+      // the listener invocation, and the completion listener 
(cleanupResources) runs
+      // without waiting.
       writerContext.markTaskFailed(writeError)
       writerContext.markTaskCompleted(Some(writeError))
 


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to