This is an automated email from the ASF dual-hosted git repository.
viirya pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 3cefd6fc8350 [SPARK-60222][CORE] Keep task completion listeners on the
completing thread when another thread invokes listeners
3cefd6fc8350 is described below
commit 3cefd6fc835018fc774f9352149ddbc259a54a6e
Author: Liang-Chi Hsieh <[email protected]>
AuthorDate: Sat Oct 31 00:14:50 2026 -0700
[SPARK-60222][CORE] Keep task completion listeners on the completing thread
when another thread invokes listeners
### What changes were proposed in this pull request?
When another thread is invoking task listeners at the moment the task
completes (e.g. a kill thread running failure listeners because an interrupt
listener threw, or a non-task thread that called `markTaskFailed` through
`Utils.tryWithSafeFinallyAndFailureCallbacks`),
`TaskContextImpl.markTaskCompleted` returns without running the completion
listeners, and the other thread only drains its own stack, so the completion
listeners never run. Failure listeners that another thread queues wh [...]
This PR keeps the completion listeners on the completing thread:
- `markTaskCompleted` claims the listener invocation in the same
synchronized block that sets `completed`. If another thread holds it,
`markTaskCompleted` registers itself as the next invoker and waits; the holder
hands the invocation over directly when it is done, so no third thread can take
it in between.
- When a listener throws, the failure listeners run nested in the current
invocation instead of clearing and reclaiming `listenerInvocationThread`, which
left a window in which another thread could take over the listeners.
- Failure listeners that another thread queues while `markTaskCompleted`
runs the completion listeners run on the completing thread before the
completion listeners that have not run yet, so failure cleanup (e.g. a state
store `abort()`) runs before the completion listener that releases the store if
that one has not run yet, also after a handover. This is the order in which a
failing completion listener already runs the failure listeners on master, and
their failures are likewise repor [...]
- The release and the handover are in a `finally`, and `markTaskCompleted`
removes a waiter or holder that it registered if an error leaves it behind.
- The handover wait keeps interrupts (they are restored once it is done)
and a failure while logging (it is thrown after the handover with the
completion listeners' failures, unless one of them is fatal, or reported after
the failures of the pre-completion listeners if there are any), and logs a
warning periodically; the first one carries the holder's stack trace, taken
outside the TaskContext lock once only, and is skipped if the holder has handed
over meanwhile. The interval is sett [...]
- A failure listener invocation that a failing listener starts runs nested,
and an error from it that `Utils.isFatalError` does not count as fatal, e.g. a
`LinkageError`, is reported with that listener's failure instead of ending the
listener loop. A fatal error, e.g. an `OutOfMemoryError`, propagates.
- `getMessage` is listener code: if it fails for a listener's exception,
the `TaskCompletionListenerException` reports that failure in place of the
exception, so that the other listeners' failures are still reported. This
applies to the completion, failure and interrupt listeners.
- The StreamingShuffleSuite test that documented the skipped completion
listener now checks that it runs on the completing thread, and the comments in
`ErrorNotifier` and `StreamingShuffleReader` are updated accordingly.
This was split out of #59332 during its review, since it changes listener
behavior for every task.
### Why are the changes needed?
Completion listeners do the task's cleanup (closing readers, releasing
state stores, freeing task-managed resources). When they are skipped, those
resources leak; when they run late on another thread, they run after the
executor has already released the task's memory. Failure listeners that are
dropped skip failure cleanup such as aborting a state store update.
Note on the wait: it has no time bound, so if the holder's listeners block,
the completing task waits too, and it now waits before the executor releases
the task's block locks and memory. On master, the task thread could release
them, but not report its result, before it blocked on the `TaskRunner` lock
that a kill thread holds while it runs `Task.kill`. So a failure listener on
another thread, e.g. a kill thread, must not wait for the task's block locks or
memory. When the holder is [...]
Out of scope: listeners added after `markTaskCompleted` has finished, while
another thread holds the invocation, are still dropped unless that thread runs
listeners of the same kind, and so are post-status-update listeners if another
thread holds the invocation when `invokePostStatusUpdateListeners` runs.
Running them would mean running them on another thread, without the task's
`TaskContext`, after the executor has released the task's block locks and
memory.
### Does this PR introduce _any_ user-facing change?
No API change. In the cases above, completion and failure listeners that
were skipped now run, and `markTaskCompleted` waits for the other thread to
finish its listeners first.
### How was this patch tested?
New tests in `TaskContextSuite`:
- a listener added from another thread as completion starts does not take
over the listeners;
- a failing listener does not let another thread take over the listeners;
- completion waits for another thread invoking listeners (a kill whose
interrupt listener throws) and runs the listeners on the completing thread;
- a listener on the invoking thread that completes the task runs the
completion listeners nested, and that thread keeps the invocation;
- a failure listener left to the completing thread runs before its next
completion listener, without and with a handover, and its failure is reported
as its own;
- an interrupt does not stop the handover wait and is kept;
- the handover wait warns with the holder's name and, on the first warning
only, its stack trace, and does not warn about a holder that has handed over;
- a failing handover warning does not end the wait: the completing thread
runs the listeners once the holder has handed over, then throws the failure
(with an appender that does not ignore its exceptions);
- a failing handover warning is reported after the failure of a
pre-completion listener, and with the failure of a completion listener, unless
the completion listener invocation throws an `OutOfMemoryError`, which then
propagates;
- a failure of the `getMessage` of a listener's exception is reported in
place of that exception, together with the other listeners' failures, for
failure listeners started by a failing pre-completion or completion listener or
left to the completing thread, for pre-completion listeners and for interrupt
listeners;
- a `LinkageError` (from a `getMessage` whose failure fails in `getMessage`
too) does not end the pre-completion, completion or left failure listener loop,
and is reported with the failure of a pre-completion listener;
- an error that escapes the listener loop (an `OutOfMemoryError` from such
a `getMessage`) still releases the invocation, and propagates instead of
becoming a suppressed exception of the listener whose failure started it;
- completion and failure listeners added after completion are not run by a
thread invoking other listeners.
Updated tests: "background markTaskFailed races with task thread
markTaskCompleted" in `StreamingShuffleSuite`, and "SPARK-56330: task
completion during interrupt listener execution" in `TaskContextSuite`, which
now completes the task on a thread that it joins with a timeout. The threads
these tests start record what they throw, and the test fails if one did.
Each behavior that these tests cover was checked by reverting it alone and
confirming that a test fails. Not covered are the allocations moved above the
claim, the cleanup in `markTaskCompleted`'s `finally`, which only an error such
as an `OutOfMemoryError` before `invokeListeners` starts can reach, the
handover when an error escapes the holder's listener loop (the loop of a
failure listener invocation does not start a nested one), and the handling of
an `InterruptedException` while w [...]
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
This pull request and its description were written by Isaac.
Closes #59663 from viirya/SPARK-60222-listener-handover.
Authored-by: Liang-Chi Hsieh <[email protected]>
Signed-off-by: Liang-Chi Hsieh <[email protected]>
(cherry picked from commit ff82f3a67e16df106ec15a970910950a35cf3112)
Signed-off-by: Liang-Chi Hsieh <[email protected]>
---
.../scala/org/apache/spark/TaskContextImpl.scala | 414 ++++++++++--
.../org/apache/spark/api/python/PythonRunner.scala | 3 +-
.../shuffle/streaming/StreamingShuffleReader.scala | 4 +-
.../org/apache/spark/util/ErrorNotifier.scala | 7 +-
.../apache/spark/scheduler/TaskContextSuite.scala | 751 ++++++++++++++++++++-
.../shuffle/streaming/StreamingShuffleSuite.scala | 109 +--
6 files changed, 1146 insertions(+), 142 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/TaskContextImpl.scala
b/core/src/main/scala/org/apache/spark/TaskContextImpl.scala
index 575d0e6f7512..45e8808812b7 100644
--- a/core/src/main/scala/org/apache/spark/TaskContextImpl.scala
+++ b/core/src/main/scala/org/apache/spark/TaskContextImpl.scala
@@ -19,6 +19,7 @@ package org.apache.spark
import java.io.Closeable
import java.util.{Properties, Stack}
+import java.util.concurrent.TimeUnit
import javax.annotation.concurrent.GuardedBy
import scala.collection.mutable.ArrayBuffer
@@ -27,7 +28,7 @@ import scala.util.control.NonFatal
import org.apache.spark.executor.TaskMetrics
import org.apache.spark.internal.{config, Logging}
-import org.apache.spark.internal.LogKeys.LISTENER
+import org.apache.spark.internal.LogKeys.{LISTENER, TASK_ATTEMPT_ID,
THREAD_NAME, TOTAL_TIME}
import org.apache.spark.memory.TaskMemoryManager
import org.apache.spark.metrics.MetricsSystem
import org.apache.spark.metrics.source.Source
@@ -63,6 +64,8 @@ private[spark] class TaskContextImpl(
extends TaskContext
with Logging {
+ import TaskContextImpl._
+
/**
* List of callback functions to execute when the task completes.
*
@@ -94,9 +97,21 @@ private[spark] class TaskContextImpl(
/**
* The thread currently executing task completion or failure listeners, if
any.
*
- * `invokeListeners()` uses this to ensure listeners are called sequentially.
+ * `invokeListeners()` uses this to ensure listeners are called
sequentially. Protected by the
+ * TaskContext lock.
+ */
+ @transient private var listenerInvocationThread: Option[Thread] = None
+
+ /**
+ * A thread waiting in `markTaskCompleted()` for another thread to stop
invoking listeners. That
+ * thread hands the listener invocation directly to this one, so that no
third thread can take
+ * it over in between. Protected by the TaskContext lock.
*/
- @transient @volatile private var listenerInvocationThread: Option[Thread] =
None
+ @transient private var nextListenerInvocationThread: Option[Thread] = None
+
+ /** How often `markTaskCompleted()` logs while it waits for another thread
invoking listeners. */
+ @transient @volatile private[spark] var listenerHandoverWarnIntervalMs: Long
=
+ DEFAULT_LISTENER_HANDOVER_WARN_INTERVAL_MS
// If defined, the corresponding task has been killed and this option
contains the reason.
@volatile private var reasonIfKilled: Option[String] = None
@@ -125,9 +140,12 @@ private[spark] class TaskContextImpl(
override def addTaskCompletionListener(listener: TaskCompletionListener):
this.type = {
val needToCallListener = synchronized {
- // If there is already a thread invoking listeners, adding the new
listener to
- // `onCompleteCallbacks` will cause that thread to execute the new
listener, and the call to
- // `invokeTaskCompletionListeners()` below will be a no-op.
+ // If there is already a thread invoking listeners, the call to
+ // `invokeTaskCompletionListeners()` below will be a no-op. If that
thread runs completion
+ // listeners (in `markTaskCompleted()`, or since it added one after
completion), or hands the
+ // invocation over to a thread waiting to do so, the new listener runs
there. Otherwise it is
+ // not run until a completion listener is added while no thread is
invoking listeners (see
+ // `invokeListeners()`).
//
// If there is no such thread, the call to
`invokeTaskCompletionListeners()` below will
// execute all listeners, including the new listener.
@@ -135,7 +153,7 @@ private[spark] class TaskContextImpl(
completed
}
if (needToCallListener) {
- invokeTaskCompletionListeners(None)
+ invokeTaskCompletionListeners(None, Claim)
}
this
}
@@ -162,7 +180,7 @@ private[spark] class TaskContextImpl(
synchronized {
onFailureCallbacks.push(listener)
failureCauseOpt
- }.foreach(invokeTaskFailureListeners)
+ }.foreach(invokeTaskFailureListeners(_, Claim))
this
}
@@ -189,7 +207,7 @@ private[spark] class TaskContextImpl(
}
private[spark] def invokePostStatusUpdateListeners(): Unit = {
- invokeListeners(onPostStatusUpdateCallbacks, "PostStatusUpdateListener",
None) {
+ invokeListeners(onPostStatusUpdateCallbacks, "PostStatusUpdateListener",
None, Claim) {
_.onStatusUpdateSent(this)
}
}
@@ -199,11 +217,19 @@ private[spark] class TaskContextImpl(
}
private[spark] override def markTaskFailed(error: Throwable): Unit = {
+ markTaskFailed(error, Claim)
+ }
+
+ /**
+ * @param mode `Nested` if this thread is already invoking listeners, i.e. a
listener failed. The
+ * failure listeners then run as part of that invocation, which
this thread keeps.
+ */
+ private def markTaskFailed(error: Throwable, mode: InvocationMode): Unit = {
synchronized {
if (failureCauseOpt.isDefined) return
failureCauseOpt = Some(error)
}
- invokeTaskFailureListeners(error)
+ invokeTaskFailureListeners(error, mode)
}
private[spark] override def markTaskCompleted(error: Option[Throwable]):
Unit = {
@@ -218,30 +244,97 @@ private[spark] class TaskContextImpl(
// writer that adds listeners lazily while it pulls its input. And since
the completion
// listeners only run after `completed` is set, they run after every
pre-completion listener,
// even if another thread is invoking listeners at this point.
+ //
+ // The completion listeners then run on this thread too, and this method
returns only after
+ // they have run. If it returned while another thread is invoking
listeners (e.g. a kill thread
+ // running failure listeners), that thread would not run them, and the
task would go on to
+ // release its memory with them pending. So the listener invocation is
claimed together with
+ // setting `completed`, which also keeps a thread that adds a completion
listener at that
+ // moment from running them; if another thread holds it, that thread hands
it over to this one
+ // when done. Failure listeners that another thread leaves to this
invocation run before the
+ // next completion listener (see `invokeListeners()`). A completion
listener added after this
+ // method has run the others runs on the thread adding it, unless another
thread is invoking
+ // listeners at that moment: that thread runs it only if it is running
completion listeners
+ // (see `invokeListeners()`).
+ //
+ // The wait for the other thread has no time bound: if its listeners
block, so does this task.
+ // It logs a warning periodically, the first one with that thread's stack
trace. The task waits
+ // here before it releases its block locks and memory (the executor does
that once `Task.run`
+ // returns), so a failure listener on another thread, e.g. a kill thread,
must not wait for
+ // either. Nothing bounds a wait for a kill thread: with
`spark.task.reaper.enabled`, every
+ // kill runs on a reaper thread, which checks
`spark.task.reaper.killTimeout` only after its
+ // kill call returns. If another thread holds the invocation and the task
is killed, the reaper
+ // with a positive kill timeout bounds the wait by killing the executor
JVM, except in local
+ // mode, where it only logs an error.
+ val currentThread = Thread.currentThread()
+ var mode: InvocationMode = Claimed
+ var awaitHandover = false
+ // Whether this call registered this thread as the listener invocation
thread or as the next
+ // one, so that it must make sure not to leave it registered.
+ var registered = false
val preCompletionExceptions = new ArrayBuffer[Throwable](2)
def nextPreCompletionListener(): Option[TaskCompletionListener] =
synchronized {
if (onPreCompleteCallbacks.empty()) {
// Set together with finding no more pre-completion listeners, so that
one added later
- // is handed to `addTaskCompletionListener` instead of being left in
the stack.
+ // is handed to `addTaskCompletionListener` instead of being left in
the stack. The
+ // listener invocation is claimed in the same step (see above).
completed = true
+ if (listenerInvocationThread.isEmpty) {
+ listenerInvocationThread = Some(currentThread)
+ registered = true
+ } else if (listenerInvocationThread.contains(currentThread)) {
+ // Called from a listener on this thread: run the completion
listeners as part of that
+ // invocation.
+ mode = Nested
+ } else {
+ nextListenerInvocationThread = Some(currentThread)
+ awaitHandover = true
+ registered = true
+ }
None
} else {
Some(onPreCompleteCallbacks.pop())
}
}
var listener: Option[TaskCompletionListener] = None
- while ({listener = nextPreCompletionListener(); listener.nonEmpty}) {
- invokeTaskPreCompletionListener(listener.get, preCompletionExceptions)
- }
-
try {
- invokeTaskCompletionListeners(error)
- } catch {
- case e: TaskCompletionListenerException if
preCompletionExceptions.nonEmpty =>
- // Merge the completion listeners' own failures, not the exception
that wraps them. Any
- // other error comes from invokeListeners itself (e.g. an
OutOfMemoryError) and is left
- // to propagate, so that the executor still sees a fatal one.
- preCompletionExceptions ++= e.getSuppressed.toSeq
+ while ({listener = nextPreCompletionListener(); listener.nonEmpty}) {
+ invokeTaskPreCompletionListener(listener.get, preCompletionExceptions)
+ }
+ try {
+ val warningFailure =
+ if (awaitHandover) awaitListenerInvocationHandover(currentThread)
else None
+ try {
+ invokeTaskCompletionListeners(error, mode)
+ } catch {
+ // The failure to warn came after the pre-completion listeners'
failures, but before
+ // those of the completion listeners. A fatal error propagates
instead.
+ case e if !Utils.isFatalError(e) && warningFailure.exists(_ ne e) =>
+ warningFailure.get.addSuppressed(e)
+ }
+ warningFailure.foreach(throw _)
+ } catch {
+ // Merge the completion listeners' own failures, not the exception
that wraps them, and
+ // any other failure that is not fatal (see `Utils.isFatalError()`),
e.g. of the warning,
+ // so that the pre-completion listeners' failures are reported first.
A fatal error (e.g.
+ // an OutOfMemoryError) propagates and ends the task's listeners, as
it ends the executor.
+ case e: TaskCompletionListenerException if
preCompletionExceptions.nonEmpty =>
+ preCompletionExceptions ++= e.getSuppressed.toSeq
+ case e if !Utils.isFatalError(e) && preCompletionExceptions.nonEmpty =>
+ preCompletionExceptions += e
+ }
+ } finally {
+ if (registered) synchronized {
+ // `invokeListeners()` releases the invocation itself once it has
started; this covers an
+ // error before that, e.g. an OutOfMemoryError. A stale waiter would
get the invocation
+ // handed to it, and a stale holder would keep every other thread from
running listeners.
+ if (nextListenerInvocationThread.contains(currentThread)) {
+ nextListenerInvocationThread = None
+ }
+ if (listenerInvocationThread.contains(currentThread)) {
+ releaseListenerInvocation()
+ }
+ }
}
if (preCompletionExceptions.nonEmpty) {
throw newTaskCompletionListenerException(preCompletionExceptions.toSeq,
error)
@@ -266,7 +359,9 @@ private[spark] class TaskContextImpl(
try {
markTaskFailed(e)
} catch {
- case t: Throwable => e.addSuppressed(t)
+ // Failures of the failure listeners. A fatal error (see
`Utils.isFatalError()`, e.g. an
+ // OutOfMemoryError) propagates and ends the task's listeners, as it
ends the executor.
+ case t if !Utils.isFatalError(t) => e.addSuppressed(t)
}
listenerExceptions += e
// Best effort: a failure to log, e.g. from an appender that does not
ignore exceptions,
@@ -283,32 +378,140 @@ private[spark] class TaskContextImpl(
private def newTaskCompletionListenerException(
listenerExceptions: Seq[Throwable],
error: Option[Throwable]): TaskCompletionListenerException = {
- val exception = new
TaskCompletionListenerException(listenerExceptions.map(_.getMessage), error)
- listenerExceptions.foreach(exception.addSuppressed)
+ // `getMessage` is listener code. Report its failure in place of the
exception it fails for:
+ // `printStackTrace`, e.g. in `Utils.exceptionString`, would fail on that
exception as well.
+ val (reported, messages) = listenerExceptions.map { e =>
+ try {
+ (e, e.getMessage)
+ } catch {
+ case t if !Utils.isFatalError(t) => (t, t.getMessage)
+ }
+ }.unzip
+ val exception = new TaskCompletionListenerException(messages, error)
+ reported.foreach(exception.addSuppressed)
exception
}
- private def invokeTaskCompletionListeners(error: Option[Throwable]): Unit = {
+ /**
+ * Waits until the thread invoking listeners hands the invocation over to
`currentThread`, which
+ * has registered itself as `nextListenerInvocationThread`.
+ *
+ * Interrupts do not stop the wait, and are passed on once it is done. The
wait of
+ * `PipelinedWriterTask.awaitExit()` in `BasePythonRunner` drops them
instead, because it is a
+ * listener itself and the listeners right after it must still be able to
clean up. This wait
+ * is not a listener but part of `markTaskCompleted()`, so it leaves the
interrupt status (e.g.
+ * set by a task kill) as the completion listeners would see it without the
wait. If the other
+ * thread is a kill thread, the interrupt of that kill does not come during
the wait: `Task.kill`
+ * interrupts the task thread once `markInterrupted()` has returned, after
the handover, so with
+ * `interruptThread` it may come while the completion listeners run here.
+ *
+ * A failure to warn (e.g. an OutOfMemoryError while logging) does not end
the wait either, like
+ * in `PipelinedWriterTask.awaitExit()`: the other thread does not run the
completion listeners,
+ * and the task would release its memory with them pending. The first such
failure is returned,
+ * for `markTaskCompleted()` to throw once it has run the completion
listeners. The wait itself
+ * is not covered, so that a lasting failure does not spin.
+ */
+ private def awaitListenerInvocationHandover(currentThread: Thread):
Option[Throwable] = {
+ val startNs = System.nanoTime()
+ var interrupted = false
+ var warned = false
+ var handedOver = false
+ var failure: Option[Throwable] = None
+ try {
+ while (!handedOver) {
+ val holder = synchronized {
+ val deadlineNs = System.nanoTime() +
+ TimeUnit.MILLISECONDS.toNanos(math.max(1L,
listenerHandoverWarnIntervalMs))
+ var remainingNs = deadlineNs - System.nanoTime()
+ while (!listenerInvocationThread.contains(currentThread) &&
remainingNs > 0) {
+ try {
+ wait(math.max(1L, TimeUnit.NANOSECONDS.toMillis(remainingNs)))
+ } catch {
+ case _: InterruptedException => interrupted = true
+ }
+ remainingNs = deadlineNs - System.nanoTime()
+ }
+ handedOver = listenerInvocationThread.contains(currentThread)
+ if (handedOver) None else listenerInvocationThread
+ }
+ // Outside the lock: walking the stack of another thread stops it at a
safepoint, and
+ // logging may block, which would otherwise hold up every thread using
this TaskContext.
+ // The holder does not change during the wait, since it hands the
invocation to this
+ // thread, so its stack trace is taken for the first warning only,
even if that fails.
+ holder match {
+ case Some(holder) =>
+ val withStackTrace = !warned
+ warned = true
+ try {
+ warnStillWaiting(holder, startNs, withStackTrace)
+ } catch {
+ // E.g. from an appender that blocks interruptibly: an interrupt
like any other.
+ case _: InterruptedException => interrupted = true
+ case t: Throwable => failure = failure.orElse(Some(t))
+ }
+ case None =>
+ }
+ }
+ } finally {
+ if (interrupted) {
+ currentThread.interrupt()
+ }
+ }
+ failure
+ }
+
+ private def warnStillWaiting(holder: Thread, startNs: Long, withStackTrace:
Boolean): Unit = {
+ val holderStack = if (withStackTrace) Some(holder.getStackTrace) else None
+ // If the holder has handed over meanwhile, its stack shows whatever it
does next.
+ if (synchronized(listenerInvocationThread.contains(holder))) {
+ val waitedMs = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startNs)
+ val message = log"Task ${MDC(TASK_ATTEMPT_ID, taskAttemptId)} has been
waiting " +
+ log"for ${MDC(TOTAL_TIME, waitedMs)} ms to run its completion
listeners until " +
+ log"thread ${MDC(THREAD_NAME, holder.getName)} finishes running its
task listeners."
+ holderStack match {
+ case Some(trace) =>
+ val stack = new Exception(s"Stack trace of thread ${holder.getName}")
+ stack.setStackTrace(trace)
+ logWarning(message, stack)
+ case None =>
+ logWarning(message)
+ }
+ }
+ }
+
+ /**
+ * Deregisters this thread as the listener invocation thread, or hands the
invocation over to a
+ * thread waiting for it in `markTaskCompleted()`. Must hold the TaskContext
lock.
+ */
+ private def releaseListenerInvocation(): Unit = {
+ listenerInvocationThread = nextListenerInvocationThread
+ nextListenerInvocationThread = None
+ notifyAll()
+ }
+
+ private def invokeTaskCompletionListeners(
+ error: Option[Throwable],
+ mode: InvocationMode): Unit = {
// It is safe to access the reference to `onCompleteCallbacks` without
holding the TaskContext
// lock. `invokeListeners()` acquires the lock before accessing the
contents.
- invokeListeners(onCompleteCallbacks, "TaskCompletionListener", error) {
+ invokeListeners(onCompleteCallbacks, "TaskCompletionListener", error,
mode) {
_.onTaskCompletion(this)
}
}
- private def invokeTaskFailureListeners(error: Throwable): Unit = {
+ private def invokeTaskFailureListeners(error: Throwable, mode:
InvocationMode): Unit = {
// It is safe to access the reference to `onFailureCallbacks` without
holding the TaskContext
// lock. `invokeListeners()` acquires the lock before accessing the
contents.
- invokeListeners(onFailureCallbacks, "TaskFailureListener", Option(error)) {
+ invokeListeners(onFailureCallbacks, "TaskFailureListener", Option(error),
mode) {
_.onTaskFailure(this, error)
}
}
private def invokeTaskInterruptListeners(reason: String, error: Throwable):
Unit = {
// Do not use `invokeListeners()`. That method uses
`listenerInvocationThread` to serialize
- // all listener invocations, which would prevent task completion or
failure listeners from
- // running if the task completes or fails while executing an interrupt
listener (causing
- // resource leaks such as task-managed resources not being freed).
+ // all listener invocations, so if the task completes or fails while an
interrupt listener
+ // runs, the task completion would wait for the interrupt listeners, and
the failure listeners
+ // would only run after them.
//
// Instead, directly execute the task interrupt listeners with independent
serialization.
// Exceptions are collected and thrown as a
TaskCompletionListenerException so that
@@ -339,17 +542,37 @@ private[spark] class TaskContextImpl(
}
}
if (listenerExceptions.nonEmpty) {
- val exception = new TaskCompletionListenerException(
- listenerExceptions.map(_.getMessage).toSeq, Option(error))
- listenerExceptions.foreach(exception.addSuppressed)
- throw exception
+ throw newTaskCompletionListenerException(listenerExceptions.toSeq,
Option(error))
}
}
+ /**
+ * Invokes and removes the listeners in `listeners`, in reverse order of
registration.
+ *
+ * Other threads cannot invoke listeners while this thread does, so they
leave the listeners
+ * they add, or whose invocation they start, meanwhile. Those in `listeners`
run here like the
+ * others. If this is the completion listener invocation of
`markTaskCompleted()`, the failure
+ * listeners left to it (the task has failed) run before the completion
listeners that have
+ * not run yet, e.g. so that a failure listener that aborts a state store
update runs before
+ * the completion listener that releases the state store, if that one has
not run yet. This is
+ * the order of a failure listener invocation that a failing completion
listener starts, and,
+ * likewise, the failures of these failure listeners are reported with those
of the completion
+ * listeners. Any other invocation does not run the listeners of another
kind that other threads
+ * left to it, since its thread may be one on which they must not run, e.g.
a kill thread, or the
+ * task thread once `Task.run` has unset its `TaskContext` and the executor
has released the
+ * task's block locks and memory. A listener that fails here still starts
the failure listeners
+ * on this thread (see `runListener`). Unless `mode` is `Nested`, this
thread releases the
+ * invocation when it is done, or hands it over to a thread waiting for it in
+ * `markTaskCompleted()`.
+ *
+ * Exceptions of the failure listeners left to this invocation are reported
to the caller of
+ * this method, together with those of `listeners`.
+ */
private def invokeListeners[T](
listeners: Stack[T],
name: String,
- error: Option[Throwable])(
+ error: Option[Throwable],
+ mode: InvocationMode)(
callback: T => Unit): Unit = {
// This method is subject to two constraints:
//
@@ -361,7 +584,16 @@ private[spark] class TaskContextImpl(
//
// We meet these constraints by ensuring there is at most one thread
invoking listeners at any
// point in time.
- synchronized {
+ //
+ // Allocated before the claim below, so that nothing can fail between the
claim and the `try`
+ // that releases it.
+ //
+ // Whether this is the completion listener invocation of
`markTaskCompleted()`, the only caller
+ // of `invokeTaskCompletionListeners()` that does not claim the invocation.
+ val completesTask = (listeners eq onCompleteCallbacks) && mode != Claim
+ val listenerExceptions = new ArrayBuffer[Throwable](2)
+ var released = mode == Nested
+ if (mode == Claim) synchronized {
if (listenerInvocationThread.nonEmpty) {
// If another thread is already invoking listeners, do nothing.
return
@@ -373,26 +605,14 @@ private[spark] class TaskContextImpl(
}
}
- def getNextListenerOrDeregisterThread(): Option[T] = synchronized {
- if (listeners.empty()) {
- // We have executed all listeners that have been added so far.
Deregister this thread as the
- // callback invocation thread.
- listenerInvocationThread = None
- None
- } else {
- Some(listeners.pop())
- }
- }
-
- val listenerExceptions = new ArrayBuffer[Throwable](2)
- var listenerOption: Option[T] = None
- while ({listenerOption = getNextListenerOrDeregisterThread();
listenerOption.nonEmpty}) {
- val listener = listenerOption.get
+ def runListener(listener: T): Unit = {
try {
callback(listener)
} catch {
case e: Throwable =>
- // A listener failed. Temporarily clear the listenerInvocationThread
and markTaskFailed.
+ // A listener failed. Call markTaskFailed, which runs the failure
listeners as part of
+ // this invocation: this thread stays registered as
`listenerInvocationThread`, so that
+ // no other thread takes over the listeners in the meantime.
//
// One of the following cases applies (#3 being the interesting one):
//
@@ -427,16 +647,11 @@ private[spark] class TaskContextImpl(
// [[markTaskFailed]] here will have no effect and we simply
resume running the
// remaining completion handlers.
try {
- listenerInvocationThread = None
- markTaskFailed(e)
+ markTaskFailed(e, Nested)
} catch {
- case t: Throwable => e.addSuppressed(t)
- } finally {
- synchronized {
- if (listenerInvocationThread.isEmpty) {
- listenerInvocationThread = Some(Thread.currentThread())
- }
- }
+ // Failures of the failure listeners. A fatal error (see
`Utils.isFatalError()`, e.g.
+ // an OutOfMemoryError) propagates, through the release below.
+ case t if !Utils.isFatalError(t) => e.addSuppressed(t)
}
listenerExceptions += e
// Best effort: a failure to log, e.g. from an appender that does
not ignore exceptions,
@@ -449,11 +664,54 @@ private[spark] class TaskContextImpl(
}
}
}
+
+ def runLeftFailureListeners(cause: Throwable): Unit = {
+ try {
+ invokeTaskFailureListeners(cause, Nested)
+ } catch {
+ // Merge the listeners' own failures, which the nested invocation has
logged, not the
+ // exception that wraps them. A fatal error (see
`Utils.isFatalError()`, e.g. an
+ // OutOfMemoryError) propagates, through the release below.
+ case e: TaskCompletionListenerException => listenerExceptions ++=
e.getSuppressed.toSeq
+ case t if !Utils.isFatalError(t) => listenerExceptions += t
+ }
+ }
+
+ // Chooses the next thing to do under the TaskContext lock, to be done
outside of it.
+ def nextStep(): Option[() => Unit] = synchronized {
+ if (completesTask && failureCauseOpt.isDefined &&
!onFailureCallbacks.empty()) {
+ val cause = failureCauseOpt.get
+ Some(() => runLeftFailureListeners(cause))
+ } else if (!listeners.empty()) {
+ val listener = listeners.pop()
+ Some(() => runListener(listener))
+ } else {
+ // We have executed all listeners that have been added so far.
Deregister this thread as
+ // the callback invocation thread, or hand the invocation over to a
waiting thread.
+ if (mode != Nested) {
+ releaseListenerInvocation()
+ }
+ None
+ }
+ }
+
+ try {
+ var step: Option[() => Unit] = None
+ while ({step = nextStep(); step.nonEmpty}) {
+ step.get()
+ }
+ released = true
+ } finally {
+ if (!released) {
+ // Something escaped the loop above. Do not leave this thread
registered, which would
+ // keep any other thread from running listeners, or a waiting thread
waiting forever.
+ synchronized {
+ releaseListenerInvocation()
+ }
+ }
+ }
if (listenerExceptions.nonEmpty) {
- val exception = new TaskCompletionListenerException(
- listenerExceptions.map(_.getMessage).toSeq, error)
- listenerExceptions.foreach(exception.addSuppressed)
- throw exception
+ throw newTaskCompletionListenerException(listenerExceptions.toSeq, error)
}
}
@@ -557,3 +815,25 @@ private[spark] class TaskContextImpl(
}
}
}
+
+private object TaskContextImpl {
+
+ /** How an `invokeListeners()` call takes part in the listener invocation. */
+ sealed trait InvocationMode
+
+ /**
+ * Registers this thread as `listenerInvocationThread` if no thread is
registered; otherwise
+ * leaves the listeners to the registered thread, which may be this one, and
which runs them only
+ * in some cases (see `invokeListeners()`).
+ */
+ case object Claim extends InvocationMode
+
+ /** The caller has already registered this thread as
`listenerInvocationThread`. */
+ case object Claimed extends InvocationMode
+
+ /** An outer invocation on this thread has registered it, and keeps it when
this one is done. */
+ case object Nested extends InvocationMode
+
+ /** How often to log while `markTaskCompleted()` waits for another thread
invoking listeners. */
+ val DEFAULT_LISTENER_HANDOVER_WARN_INTERVAL_MS = 10000L
+}
diff --git a/core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala
b/core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala
index 84109d042540..f0a918cf8637 100644
--- a/core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala
+++ b/core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala
@@ -378,7 +378,8 @@ private[spark] object BasePythonRunner extends Logging {
// An interrupt of this thread that arrives during the wait (e.g. a task
kill) does not stop
// it and is not passed on (see stopAndAwaitExit): the kill is already
recorded in the
// TaskContext, and the flag would make the blocking channel and lock
operations of the
- // completion listeners that run next fail and skip cleanup.
+ // completion listeners that run next fail and skip cleanup (see also
+ // TaskContextImpl.awaitListenerInvocationHandover, which passes
interrupts on).
val intervalNs = TimeUnit.MILLISECONDS.toNanos(warnIntervalMs)
val startNs = System.nanoTime()
var nextWarningNs = startNs + intervalNs
diff --git
a/core/src/main/scala/org/apache/spark/shuffle/streaming/StreamingShuffleReader.scala
b/core/src/main/scala/org/apache/spark/shuffle/streaming/StreamingShuffleReader.scala
index 70ea96154344..160a01794d32 100644
---
a/core/src/main/scala/org/apache/spark/shuffle/streaming/StreamingShuffleReader.scala
+++
b/core/src/main/scala/org/apache/spark/shuffle/streaming/StreamingShuffleReader.scala
@@ -400,8 +400,8 @@ class StreamingShuffleReader[K, C](
private def checkTaskFailure(): Unit = {
context.getTaskFailure.foreach(throw _)
// Surface any background error on the task thread so that markTaskFailed
and
- // markTaskCompleted are called from this thread, ensuring completion
listeners
- // (including cleanupResources) run without contention.
+ // markTaskCompleted are called from this thread, so that the failure and
completion
+ // listeners (including cleanupResources) run here without waiting for
another thread.
errorNotifier.throwErrorIfExists()
if (context.isInterrupted()) {
throw new InterruptedException("Task interrupted. Exiting read loop.")
diff --git a/core/src/main/scala/org/apache/spark/util/ErrorNotifier.scala
b/core/src/main/scala/org/apache/spark/util/ErrorNotifier.scala
index a40e0c7bd524..5660ffb427c6 100644
--- a/core/src/main/scala/org/apache/spark/util/ErrorNotifier.scala
+++ b/core/src/main/scala/org/apache/spark/util/ErrorNotifier.scala
@@ -28,9 +28,10 @@ import org.apache.spark.internal.Logging
*
* A background thread records the first error via [[markError]] rather than
failing the task
* directly; the owning thread polls [[throwErrorIfExists]] (or [[getError]])
at a safe point and
- * re-throws on its own thread. This avoids a race where calling
`TaskContext.markTaskFailed` from
- * a background thread can run concurrently with the task thread's
`markTaskCompleted` and silently
- * skip completion listeners (including cleanup), leading to thread leaks.
+ * re-throws on its own thread. Calling `TaskContext.markTaskFailed` from a
background thread
+ * can instead run the failure listeners on that thread, if no other thread is
invoking listeners,
+ * and a concurrent `markTaskCompleted` on the task thread then waits for them
before it runs the
+ * completion listeners (including cleanup).
*
* Shared by the streaming shuffle and the streaming-query async checkpointing
paths.
*/
diff --git
a/core/src/test/scala/org/apache/spark/scheduler/TaskContextSuite.scala
b/core/src/test/scala/org/apache/spark/scheduler/TaskContextSuite.scala
index bc4e6cbe76ce..f1454ac03b5c 100644
--- a/core/src/test/scala/org/apache/spark/scheduler/TaskContextSuite.scala
+++ b/core/src/test/scala/org/apache/spark/scheduler/TaskContextSuite.scala
@@ -22,14 +22,17 @@ import java.util.concurrent.{ConcurrentLinkedQueue,
CountDownLatch, Semaphore, T
import java.util.concurrent.atomic.{AtomicInteger, AtomicReference}
import scala.collection.mutable.ArrayBuffer
+import scala.concurrent.duration._
import scala.jdk.CollectionConverters._
+import org.apache.logging.log4j.Level
import org.apache.logging.log4j.core.LogEvent
import org.apache.logging.log4j.core.appender.AbstractAppender
import org.apache.logging.log4j.core.config.Property
import org.mockito.ArgumentMatchers.{any, eq => mockitoEq}
import org.mockito.Mockito._
import org.scalatest.BeforeAndAfter
+import org.scalatest.concurrent.Eventually.{eventually, timeout}
import org.apache.spark._
import org.apache.spark.api.plugin.{DriverPlugin, ExecutorPlugin, SparkPlugin}
@@ -398,23 +401,22 @@ class TaskContextSuite extends SparkFunSuite with
BeforeAndAfter with LocalSpark
})
// Interrupt the task from a separate thread and wait until the interrupt
listener starts.
- val interruptThread = new Thread(() => context.markInterrupted("test
interrupt"))
- interruptThread.start()
- interruptListenerStarted.acquire()
-
- // While the interrupt listener is running on the interrupt thread, mark
the task completed
- // on this thread. With the dedicated interrupt listener loop, this must
NOT be blocked.
- context.markTaskCompleted(None)
-
- // The completion listener should have been called even though the
interrupt listener is still
- // running. If `invokeListeners()` were shared between interrupt and
completion listeners,
- // the completion listener would be silently skipped because
`listenerInvocationThread` would
- // be held by the interrupt thread.
- verify(completionListener, times(1)).onTaskCompletion(any())
-
- // Release the interrupt listener and join the interrupt thread.
- interruptListenerRelease.release()
- interruptThread.join()
+ val interruptThread =
startDaemonThread("interrupt")(context.markInterrupted("test interrupt"))
+ try {
+ interruptListenerStarted.acquire()
+
+ // While the interrupt listener is running on the interrupt thread, mark
the task completed
+ // on another thread. With the dedicated interrupt listener loop, this
must NOT be blocked.
+ // If `invokeListeners()` were shared between interrupt and completion
listeners, the
+ // interrupt thread would hold `listenerInvocationThread`, and the
completion would wait
+ // for it, so the join below would time out.
+
joinThread(startDaemonThread("completion")(context.markTaskCompleted(None)))
+ verify(completionListener, times(1)).onTaskCompletion(any())
+ } finally {
+ // Release the interrupt listener and join the interrupt thread.
+ interruptListenerRelease.release()
+ }
+ joinThread(interruptThread)
}
test("FailureListener throws after task body fails") {
@@ -858,14 +860,9 @@ class TaskContextSuite extends SparkFunSuite with
BeforeAndAfter with LocalSpark
// Create a listener that registers another listener using a second thread.
val multithreadedListener = new TaskCompletionListener {
override def onTaskCompletion(context: TaskContext): Unit = {
- val thread = new Thread(new Runnable {
- override def run(): Unit = {
- context.addTaskCompletionListener(simpleListener)
- }
- })
- thread.start()
+ val thread =
startDaemonThread("adder")(context.addTaskCompletionListener(simpleListener))
invocations.getAndIncrement()
- thread.join()
+ joinThread(thread)
}
}
context.addTaskCompletionListener(multithreadedListener)
@@ -1195,19 +1192,721 @@ class TaskContextSuite extends SparkFunSuite with
BeforeAndAfter with LocalSpark
completion
}
+ /** A daemon thread that records what `body` throws, for `joinThread` to
check. */
+ private class DaemonThread(name: String, body: => Unit) extends Thread(name)
{
+ setDaemon(true)
+ @volatile var error: Throwable = _
+ override def run(): Unit = {
+ try {
+ body
+ } catch {
+ case t: Throwable => error = t
+ }
+ }
+ }
+
+ private def startDaemonThread(name: String)(body: => Unit): DaemonThread = {
+ val thread = new DaemonThread(name, body)
+ thread.start()
+ thread
+ }
+
+ private def joinThread(thread: DaemonThread): Unit = {
+ thread.join(TimeUnit.MINUTES.toMillis(1))
+ assert(!thread.isAlive, s"${thread.getName} is still running")
+ Option(thread.error).foreach(e => fail(s"${thread.getName} failed", e))
+ }
+
+ /**
+ * Runs `takeInvocation` on a new thread named `name`, by default failing
the task, and waits
+ * until a failure listener on it holds the listener invocation, which it
does until the
+ * returned latch is counted down. Returns that latch and the thread.
+ */
+ private def holdListenerInvocation(context: TaskContext, name: String =
"listener-holder")(
+ takeInvocation: => Unit = context.markTaskFailed(new Exception("failed
on another thread")))
+ : (CountDownLatch, DaemonThread) = {
+ val holding = new CountDownLatch(1)
+ val release = new CountDownLatch(1)
+ context.addTaskFailureListener(new TaskFailureListener {
+ override def onTaskFailure(context: TaskContext, error: Throwable): Unit
= {
+ holding.countDown()
+ release.await(1, TimeUnit.MINUTES)
+ }
+ })
+ val holder = startDaemonThread(name)(takeInvocation)
+ assert(holding.await(1, TimeUnit.MINUTES))
+ (release, holder)
+ }
+
/**
- * An appender that throws `failure` for every event whose message contains
`trigger`. Unlike
- * `LogAppender`, it does not ignore exceptions, so log4j throws them from
the logging call.
+ * Fails every logging call whose message contains `trigger` with `failure`.
Unlike
+ * `LogAppender`, it does not ignore its exceptions, so log4j passes them to
the caller.
*/
private class FailingAppender(trigger: String, failure: RuntimeException)
extends AbstractAppender("failingAppender", null, null, false,
Property.EMPTY_ARRAY) {
+ val failures = new AtomicInteger()
+ // Of these, the failures of events that carry a throwable.
+ val failuresWithThrown = new AtomicInteger()
override def append(event: LogEvent): Unit = {
if (event.getMessage.getFormattedMessage.contains(trigger)) {
+ failures.incrementAndGet()
+ if (event.getThrown != null) failuresWithThrown.incrementAndGet()
throw failure
}
}
}
+ private def isCausedBy(error: Throwable, cause: Throwable): Boolean =
+ Iterator.iterate(error)(_.getCause).takeWhile(_ != null).exists(_ eq cause)
+
+ /** An exception whose `getMessage`, which is listener code too, throws
`failure`. */
+ private class MessageFailingException(failure: Throwable) extends Exception {
+ override def getMessage: String = throw failure
+ }
+
+ private def addFailureListenerThrowing(context: TaskContext)(failure: =>
Throwable): Unit =
+ context.addTaskFailureListener(new TaskFailureListener {
+ override def onTaskFailure(context: TaskContext, error: Throwable): Unit
= throw failure
+ })
+
+ /** Waits until `thread` waits in `markTaskCompleted()` for the listener
invocation. */
+ private def awaitHandoverWait(thread: Thread): Unit = {
+ eventually(timeout(1.minute)) {
+ assert(!thread.isAlive || thread.getState == Thread.State.TIMED_WAITING)
+ }
+ assert(thread.isAlive, s"${thread.getName} returned instead of waiting")
+ }
+
+ test("SPARK-60222: a listener added as completion starts does not take over
the listeners") {
+ // A thread that sees the task completed and adds a listener at that
moment must not become
+ // the thread that runs the pending listeners: the completing thread
claims the listener
+ // invocation together with setting `completed`.
+ (1 to 500).foreach { _ =>
+ val context = TaskContext.empty()
+ val pendingListenerThread = new AtomicReference[Thread]()
+ context.addTaskCompletionListener[Unit] { _ =>
+ pendingListenerThread.set(Thread.currentThread())
+ }
+ val ready = new CountDownLatch(1)
+ val adder = startDaemonThread("adder") {
+ ready.countDown()
+ while (!context.isCompleted()) {}
+ context.addTaskCompletionListener[Unit] { _ => }
+ }
+ ready.await()
+ context.markTaskCompleted(None)
+ joinThread(adder)
+ assert(pendingListenerThread.get eq Thread.currentThread())
+ }
+ }
+
+ test("SPARK-60222: a failing listener does not let another thread take over
the listeners") {
+ // While a failing listener's failure listeners run, the thread invoking
the listeners must
+ // stay registered, so that another thread adding listeners at that moment
does not start
+ // running the pending ones, possibly concurrently with this thread.
+ (1 to 200).foreach { _ =>
+ val context = TaskContext.empty()
+ val pendingListenerThread = new AtomicReference[Thread]()
+ context.addTaskCompletionListener[Unit] { _ =>
+ pendingListenerThread.set(Thread.currentThread())
+ }
+ // Two failing listeners, which run first: the second failure finds the
task already failed.
+ (1 to 2).foreach { i =>
+ context.addTaskCompletionListener(new TaskCompletionListener {
+ override def onTaskCompletion(context: TaskContext): Unit = throw
new Exception(s"$i")
+ })
+ }
+ context.addTaskFailureListener(new TaskFailureListener {
+ override def onTaskFailure(context: TaskContext, error: Throwable):
Unit = {}
+ })
+ val ready = new CountDownLatch(1)
+ val adder = startDaemonThread("adder") {
+ ready.countDown()
+ while (!context.isCompleted()) {}
+ (1 to 200).foreach { _ => context.addTaskCompletionListener[Unit] { _
=> } }
+ }
+ ready.await()
+ intercept[TaskCompletionListenerException] {
+ context.markTaskCompleted(None)
+ }
+ joinThread(adder)
+ assert(pendingListenerThread.get eq Thread.currentThread())
+ }
+ }
+
+ test("SPARK-60222: completion waits for another thread invoking listeners
and runs them") {
+ val context = TaskContext.empty()
+ val listenerThreads = new ConcurrentLinkedQueue[(String, Thread)]()
+ context.addTaskCompletionListener(loggingListener(listenerThreads,
"pending"))
+ context.addTaskInterruptListener(new TaskInterruptListener {
+ override def onTaskInterrupted(context: TaskContext, reason: String):
Unit =
+ throw new Exception("interrupt listener")
+ })
+ // Like a kill: an interrupt listener fails, so the kill thread runs the
failure listeners.
+ val (release, killer) = holdListenerInvocation(context, "killer") {
+ context.markInterrupted("killed")
+ }
+
+ val completion =
startDaemonThread("completion")(context.markTaskCompleted(None))
+ try {
+ awaitHandoverWait(completion)
+ // Neither returns early nor runs the completion listeners while the
kill thread holds the
+ // listener invocation, and a third thread adding a listener does not
take them over.
+ joinThread(startDaemonThread("adder") {
+ context.addTaskCompletionListener(loggingListener(listenerThreads,
"added"))
+ })
+ assert(completion.isAlive)
+ assert(listenerThreads.isEmpty)
+ } finally {
+ release.countDown()
+ }
+ joinThread(completion)
+ joinThread(killer)
+ assert(listenerThreads.asScala.toSeq === Seq(("added", completion),
("pending", completion)))
+ }
+
+ test("SPARK-60222: a listener on the invoking thread completing the task
runs them nested") {
+ val context = TaskContext.empty()
+ val completionListenerThread = new AtomicReference[Thread]()
+ context.addTaskCompletionListener[Unit] { _ =>
+ completionListenerThread.set(Thread.currentThread())
+ }
+ val ranBeforeReturn = new AtomicReference[Option[Boolean]](None)
+ val laterListenerThread = new AtomicReference[Thread]()
+ context.addTaskFailureListener(new TaskFailureListener {
+ override def onTaskFailure(context: TaskContext, error: Throwable): Unit
= {
+ // This thread holds the listener invocation, so it must not wait for
itself.
+ context.markTaskCompleted(Some(error))
+ ranBeforeReturn.set(Some(completionListenerThread.get eq
Thread.currentThread()))
+ // And it still holds the invocation, so another thread adding a
listener does not run it.
+ joinThread(startDaemonThread("adder") {
+ context.addTaskCompletionListener[Unit] { _ =>
+ laterListenerThread.set(Thread.currentThread())
+ }
+ })
+ }
+ })
+ joinThread(startDaemonThread("failer")(context.markTaskFailed(new
Exception("failed"))))
+ assert(ranBeforeReturn.get === Some(true))
+ assert(laterListenerThread.get == null)
+ }
+
+ test("SPARK-60222: a failure listener left to the completing thread runs
before the next " +
+ "completion listener") {
+ val context = TaskContext.empty()
+ val invocations = new ConcurrentLinkedQueue[(String, Thread)]()
+ context.addTaskFailureListener(new TaskFailureListener {
+ override def onTaskFailure(context: TaskContext, error: Throwable): Unit
= {
+ invocations.add(("failure", Thread.currentThread()))
+ throw new Exception("failure listener")
+ }
+ })
+ // E.g. a completion listener that releases a state store, which a failure
listener aborts.
+ context.addTaskCompletionListener(loggingListener(invocations, "second"))
+ context.addTaskCompletionListener[Unit] { _ =>
+ invocations.add(("first", Thread.currentThread()))
+ // Another thread fails the task while this thread runs the completion
listeners, e.g. a
+ // kill whose interrupt listener failed. It finds the listener
invocation taken and leaves
+ // the failure listeners to this thread.
+ joinThread(startDaemonThread("failer") {
+ context.markTaskFailed(new Exception("failed elsewhere"))
+ })
+ assert(invocations.size == 1)
+ }
+
+ val error = intercept[TaskCompletionListenerException] {
+ context.markTaskCompleted(None)
+ }
+ val current = Thread.currentThread()
+ assert(invocations.asScala.toSeq ===
+ Seq(("first", current), ("failure", current), ("second", current)))
+ // The failure listener's failure is reported as its own, not wrapped in
the exception of the
+ // failure listener invocation.
+ assert(error.getSuppressed.map(_.getMessage).toSeq === Seq("failure
listener"))
+ }
+
+ test("SPARK-60222: a failure listener left to the thread the invocation was
handed over to " +
+ "runs before its next completion listener") {
+ val context = TaskContext.empty()
+ val invocations = new ConcurrentLinkedQueue[(String, Thread)]()
+ context.addTaskCompletionListener(loggingListener(invocations, "second"))
+ context.addTaskCompletionListener[Unit] { _ =>
+ invocations.add(("first", Thread.currentThread()))
+ // The task has failed, so the added failure listener is to run right
away. This thread
+ // holds the listener invocation, so the adding thread leaves it to this
one.
+ joinThread(startDaemonThread("adder") {
+ context.addTaskFailureListener(new TaskFailureListener {
+ override def onTaskFailure(context: TaskContext, error: Throwable):
Unit =
+ invocations.add(("failure", Thread.currentThread()))
+ })
+ })
+ assert(invocations.size == 1)
+ }
+ val (release, holder) = holdListenerInvocation(context)()
+ val completion =
startDaemonThread("completion")(context.markTaskCompleted(None))
+ try {
+ awaitHandoverWait(completion)
+ } finally {
+ release.countDown()
+ }
+ joinThread(completion)
+ joinThread(holder)
+ assert(invocations.asScala.toSeq ===
+ Seq(("first", completion), ("failure", completion), ("second",
completion)))
+ }
+
+ test("SPARK-60222: a failure of getMessage is reported in place of the
exception it fails for") {
+ val messageFailure = new IllegalStateException("no message")
+ val current = Thread.currentThread()
+ // The failure listener invocation that a failing pre-completion or
completion listener
+ // starts reports the failure of `getMessage`, and the listener loops go
on.
+ Seq(true, false).foreach { preCompletion =>
+ val context = TaskContext.empty()
+ val invocations = new ConcurrentLinkedQueue[(String, Thread)]()
+ addFailureListenerThrowing(context)(new
MessageFailingException(messageFailure))
+ context.addTaskCompletionListener(loggingListener(invocations, "next"))
+ val failing = new TaskCompletionListener {
+ override def onTaskCompletion(context: TaskContext): Unit = throw new
Exception("listener")
+ }
+ if (preCompletion) {
+ context.addTaskPreCompletionListener(failing)
+ } else {
+ context.addTaskCompletionListener(failing)
+ }
+ val error = intercept[TaskCompletionListenerException] {
+ context.markTaskCompleted(None)
+ }
+ assert(invocations.asScala.toSeq === Seq(("next", current)),
s"preCompletion=$preCompletion")
+ assert(context.isCompleted())
+ val listenerFailure = error.getSuppressed.toSeq.head
+ assert(listenerFailure.getMessage === "listener")
+ assert(listenerFailure.getSuppressed.toSeq.flatMap(_.getSuppressed) ===
Seq(messageFailure))
+ }
+
+ // The failures of the other listeners are kept: of the failure listeners
left to the
+ // completing thread,
+ {
+ val context = TaskContext.empty()
+ val invocations = new ConcurrentLinkedQueue[(String, Thread)]()
+ val otherFailure = new Exception("a")
+ addFailureListenerThrowing(context)(otherFailure)
+ addFailureListenerThrowing(context)(new
MessageFailingException(messageFailure))
+ context.addTaskCompletionListener(loggingListener(invocations, "second"))
+ context.addTaskCompletionListener[Unit] { _ =>
+ joinThread(startDaemonThread("failer") {
+ context.markTaskFailed(new Exception("failed elsewhere"))
+ })
+ }
+ val error = intercept[TaskCompletionListenerException] {
+ context.markTaskCompleted(None)
+ }
+ assert(invocations.asScala.toSeq === Seq(("second", current)))
+ assert(error.getSuppressed.toSeq === Seq(messageFailure, otherFailure))
+ assert(error.getMessage.contains("no message") &&
error.getMessage.contains("a"))
+ }
+
+ // of the pre-completion listeners,
+ {
+ val context = TaskContext.empty()
+ val otherFailure = new Exception("a")
+ Seq(otherFailure, new MessageFailingException(messageFailure)).foreach {
failure =>
+ context.addTaskPreCompletionListener(new TaskCompletionListener {
+ override def onTaskCompletion(context: TaskContext): Unit = throw
failure
+ })
+ }
+ val error = intercept[TaskCompletionListenerException] {
+ context.markTaskCompleted(None)
+ }
+ assert(context.isCompleted())
+ assert(error.getSuppressed.toSeq === Seq(messageFailure, otherFailure))
+ }
+
+ // and of the interrupt listeners, whose failure is recorded instead of
escaping the kill.
+ {
+ val context = TaskContext.empty()
+ val otherFailure = new Exception("a")
+ Seq(otherFailure, new MessageFailingException(messageFailure)).foreach {
failure =>
+ context.addTaskInterruptListener(new TaskInterruptListener {
+ override def onTaskInterrupted(context: TaskContext, reason:
String): Unit =
+ throw failure
+ })
+ }
+ context.markInterrupted("killed")
+ val recorded = context.getTaskFailure
+ assert(recorded.exists(_.isInstanceOf[TaskCompletionListenerException]),
recorded)
+ assert(recorded.get.getSuppressed.toSeq.take(2) === Seq(messageFailure,
otherFailure))
+ }
+ }
+
+ test("SPARK-60222: an error that is not fatal to the executor does not end
the listener loops") {
+ // A `LinkageError` is not `NonFatal`, but the executor keeps running. It
escapes the failure
+ // listener invocation if the failure of a `getMessage` fails in
`getMessage` too.
+ val linkageError = new LinkageError("no message")
+ def failure: Throwable =
+ new MessageFailingException(new MessageFailingException(linkageError))
+ val current = Thread.currentThread()
+ // After a failing pre-completion or completion listener,
+ Seq(true, false).foreach { preCompletion =>
+ val context = TaskContext.empty()
+ val invocations = new ConcurrentLinkedQueue[(String, Thread)]()
+ addFailureListenerThrowing(context)(failure)
+ context.addTaskCompletionListener(loggingListener(invocations, "next"))
+ val failing = new TaskCompletionListener {
+ override def onTaskCompletion(context: TaskContext): Unit = throw new
Exception("listener")
+ }
+ if (preCompletion) {
+ context.addTaskPreCompletionListener(failing)
+ } else {
+ context.addTaskCompletionListener(failing)
+ }
+ val error = intercept[TaskCompletionListenerException] {
+ context.markTaskCompleted(None)
+ }
+ assert(invocations.asScala.toSeq === Seq(("next", current)),
s"preCompletion=$preCompletion")
+ assert(context.isCompleted())
+ val listenerFailure = error.getSuppressed.toSeq.head
+ assert(listenerFailure.getMessage === "listener")
+ assert(listenerFailure.getSuppressed.exists(_ eq linkageError))
+ }
+
+ // For the failure listeners left to the completing thread,
+ {
+ val context = TaskContext.empty()
+ val invocations = new ConcurrentLinkedQueue[(String, Thread)]()
+ addFailureListenerThrowing(context)(failure)
+ context.addTaskCompletionListener(loggingListener(invocations, "second"))
+ context.addTaskCompletionListener[Unit] { _ =>
+ joinThread(startDaemonThread("failer") {
+ context.markTaskFailed(new Exception("failed elsewhere"))
+ })
+ }
+ val error = intercept[TaskCompletionListenerException] {
+ context.markTaskCompleted(None)
+ }
+ assert(invocations.asScala.toSeq === Seq(("second", current)))
+ assert(error.getSuppressed.toSeq === Seq(linkageError))
+ }
+
+ // and when the completion listener invocation throws it after a
pre-completion listener
+ // failed, it is reported with that failure.
+ val context = TaskContext.empty()
+ val preCompletionFailure = new Exception("pre-completion listener")
+ context.addTaskPreCompletionListener(new TaskCompletionListener {
+ override def onTaskCompletion(context: TaskContext): Unit = throw
preCompletionFailure
+ })
+ context.addTaskCompletionListener[Unit] { _ => throw failure }
+ val error = intercept[TaskCompletionListenerException] {
+ context.markTaskCompleted(None)
+ }
+ assert(error.getSuppressed.toSeq === Seq(preCompletionFailure,
linkageError))
+ }
+
+ test("SPARK-60222: an error that escapes the listener loop still releases
the invocation") {
+ val context = TaskContext.empty()
+ context.markTaskCompleted(None)
+ val fatal = new OutOfMemoryError("no message")
+ addFailureListenerThrowing(context)(new MessageFailingException(fatal))
+ val error = intercept[OutOfMemoryError] {
+ // Runs right away on this thread and fails, and the failure listener
invocation that this
+ // starts throws `fatal` while building its exception.
+ context.addTaskCompletionListener[Unit] { _ => throw new
Exception("listener") }
+ }
+ assert(error eq fatal)
+ // Released: a listener added now runs right away on this thread.
+ val laterListenerThread = new AtomicReference[Thread]()
+ context.addTaskCompletionListener[Unit] { _ =>
+ laterListenerThread.set(Thread.currentThread())
+ }
+ assert(laterListenerThread.get eq Thread.currentThread())
+ }
+
+ test("SPARK-60222: a fatal error of a failure listener invocation is not
hidden by the failure " +
+ "of the listener that started it") {
+ val fatal = new OutOfMemoryError("no message")
+ // Both for a failing completion listener and for a failing pre-completion
listener.
+ Seq(true, false).foreach { preCompletion =>
+ val context = TaskContext.empty()
+ addFailureListenerThrowing(context)(new MessageFailingException(fatal))
+ val failing = new TaskCompletionListener {
+ override def onTaskCompletion(context: TaskContext): Unit = throw new
Exception("listener")
+ }
+ if (preCompletion) {
+ context.addTaskPreCompletionListener(failing)
+ } else {
+ context.addTaskCompletionListener(failing)
+ }
+ // It propagates, e.g. so that the executor sees it if the task body
succeeded, instead of
+ // being a suppressed exception of the listener's failure.
+ val error = intercept[OutOfMemoryError] {
+ context.markTaskCompleted(None)
+ }
+ assert(error eq fatal, s"preCompletion=$preCompletion")
+ }
+ }
+
+ test("SPARK-60222: an interrupt does not stop the handover wait and is
kept") {
+ val context = TaskContext.empty()
+ val listenerThreads = new ConcurrentLinkedQueue[(String, Thread)]()
+ (1 to 2).foreach { i =>
+ context.addTaskCompletionListener(loggingListener(listenerThreads,
s"completion $i"))
+ }
+ val (release, holder) = holdListenerInvocation(context)()
+ val interruptedAfterCompletion = new AtomicReference[Option[Boolean]](None)
+ val completion = startDaemonThread("completion") {
+ context.markTaskCompleted(None)
+
interruptedAfterCompletion.set(Some(Thread.currentThread().isInterrupted))
+ }
+ try {
+ awaitHandoverWait(completion)
+ completion.interrupt()
+ // Still waiting for the holder, without having run any completion
listener.
+ eventually(timeout(1.minute)) {
+ assert(!completion.isAlive ||
+ !completion.isInterrupted && completion.getState ==
Thread.State.TIMED_WAITING)
+ }
+ assert(completion.isAlive, "completion returned instead of waiting")
+ assert(listenerThreads.isEmpty)
+ } finally {
+ release.countDown()
+ }
+ joinThread(completion)
+ joinThread(holder)
+ assert(interruptedAfterCompletion.get === Some(true))
+ assert(listenerThreads.asScala.toSeq ===
+ Seq(("completion 2", completion), ("completion 1", completion)))
+ }
+
+ test("SPARK-60222: the handover wait warns with the holder's name and stack
trace") {
+ val context = TaskContext.empty()
+ context.listenerHandoverWarnIntervalMs = 100
+ val (release, holder) = holdListenerInvocation(context)()
+ val logAppender = new LogAppender("listener handover wait")
+ var completion: DaemonThread = null
+ try {
+ withLogAppender(logAppender, Seq(classOf[TaskContextImpl].getName),
Some(Level.WARN)) {
+ completion =
startDaemonThread("completion")(context.markTaskCompleted(None))
+ eventually(timeout(1.minute)) {
+ assert(!completion.isAlive ||
+ logAppender.loggingEvents.count(_.getLevel == Level.WARN) >= 2)
+ }
+ assert(completion.isAlive, "completion returned instead of waiting")
+ }
+ } finally {
+ release.countDown()
+ }
+ joinThread(completion)
+ joinThread(holder)
+ val warnings = logAppender.loggingEvents.filter(_.getLevel ==
Level.WARN).toSeq
+ warnings.foreach { event =>
+ assert(event.getMessage.getFormattedMessage.contains("until thread
listener-holder"))
+ }
+ // The first warning carries the holder's stack trace, blocked in its
failure listener.
+ val holderStack = warnings.head.getThrown
+ assert(holderStack.getMessage.contains("listener-holder"))
+ assert(holderStack.getStackTrace.exists(_.getMethodName ==
"onTaskFailure"))
+ // The later ones do not walk the stack again.
+ warnings.tail.foreach { event => assert(event.getThrown == null) }
+ }
+
+ test("SPARK-60222: a failing handover warning does not end the wait") {
+ val context = TaskContext.empty()
+ context.listenerHandoverWarnIntervalMs = 50
+ val listenerThreads = new ConcurrentLinkedQueue[(String, Thread)]()
+ context.addTaskCompletionListener(loggingListener(listenerThreads,
"pending"))
+ val (release, holder) = holdListenerInvocation(context)()
+ val logFailure = new IllegalStateException("cannot log")
+ val appender = new FailingAppender("has been waiting", logFailure)
+ val completionError = new AtomicReference[Throwable]()
+ var completion: DaemonThread = null
+ try {
+ withLogAppender(appender, Seq(classOf[TaskContextImpl].getName),
Some(Level.WARN)) {
+ completion = startDaemonThread("completion") {
+ try {
+ context.markTaskCompleted(None)
+ } catch {
+ case t: Throwable => completionError.set(t)
+ }
+ }
+ eventually(timeout(1.minute)) {
+ assert(!completion.isAlive || appender.failures.get >= 2)
+ }
+ assert(completion.isAlive, "completion returned instead of waiting")
+ assert(listenerThreads.isEmpty)
+ }
+ } finally {
+ release.countDown()
+ }
+ joinThread(completion)
+ joinThread(holder)
+ // The completing thread runs the listeners once the holder has handed
over, and then throws
+ // the first failure.
+ assert(listenerThreads.asScala.toSeq === Seq(("pending", completion)))
+ assert(isCausedBy(completionError.get, logFailure))
+ // Only the first attempt to warn took the holder's stack trace, although
it failed.
+ assert(appender.failuresWithThrown.get === 1)
+ // And it has released the invocation: a listener added now runs right
away on this thread.
+ val laterListenerThread = new AtomicReference[Thread]()
+ context.addTaskCompletionListener[Unit] { _ =>
+ laterListenerThread.set(Thread.currentThread())
+ }
+ assert(laterListenerThread.get eq Thread.currentThread())
+ }
+
+ test("SPARK-60222: a failing handover warning is reported with the failures
of the " +
+ "completion listeners, unless one of them is fatal") {
+ // `fatal` is whether the completion listener invocation throws an
OutOfMemoryError, while
+ // building its exception.
+ Seq(false, true).foreach { fatal =>
+ val context = TaskContext.empty()
+ context.listenerHandoverWarnIntervalMs = 50
+ val oom = new OutOfMemoryError("no message")
+ val listenerFailure =
+ if (fatal) new MessageFailingException(oom) else new
Exception("completion listener")
+ context.addTaskCompletionListener[Unit] { _ => throw listenerFailure }
+ val (release, holder) = holdListenerInvocation(context)()
+ val logFailure = new IllegalStateException("cannot log")
+ val appender = new FailingAppender("has been waiting", logFailure)
+ val completionError = new AtomicReference[Throwable]()
+ var completion: DaemonThread = null
+ try {
+ withLogAppender(appender, Seq(classOf[TaskContextImpl].getName),
Some(Level.WARN)) {
+ completion = startDaemonThread("completion") {
+ try {
+ context.markTaskCompleted(None)
+ } catch {
+ case t: Throwable => completionError.set(t)
+ }
+ }
+ eventually(timeout(1.minute)) {
+ assert(!completion.isAlive || appender.failures.get >= 1)
+ }
+ }
+ } finally {
+ release.countDown()
+ }
+ joinThread(completion)
+ joinThread(holder)
+ val error = completionError.get
+ if (fatal) {
+ // It propagates instead of becoming a suppressed exception of the
failure to warn.
+ assert(error eq oom)
+ } else {
+ // The failure to warn, with the completion listeners' failure.
+ assert(isCausedBy(error, logFailure), error)
+ val reported = error.getSuppressed.toSeq
+ assert(reported.exists {
+ case e: TaskCompletionListenerException => e.getSuppressed.exists(_
eq listenerFailure)
+ case _ => false
+ }, reported)
+ }
+ }
+ }
+
+ test("SPARK-60222: a failing handover warning is reported with the failures
of the " +
+ "pre-completion listeners") {
+ val context = TaskContext.empty()
+ context.listenerHandoverWarnIntervalMs = 50
+ val preCompletionFailure = new Exception("pre-completion listener")
+ context.addTaskPreCompletionListener(new TaskCompletionListener {
+ override def onTaskCompletion(context: TaskContext): Unit = throw
preCompletionFailure
+ })
+ val (release, holder) = holdListenerInvocation(context)()
+ val logFailure = new IllegalStateException("cannot log")
+ val appender = new FailingAppender("has been waiting", logFailure)
+ val completionError = new AtomicReference[Throwable]()
+ var completion: DaemonThread = null
+ try {
+ withLogAppender(appender, Seq(classOf[TaskContextImpl].getName),
Some(Level.WARN)) {
+ completion = startDaemonThread("completion") {
+ try {
+ context.markTaskCompleted(None)
+ } catch {
+ case t: Throwable => completionError.set(t)
+ }
+ }
+ eventually(timeout(1.minute)) {
+ assert(!completion.isAlive || appender.failures.get >= 1)
+ }
+ }
+ } finally {
+ release.countDown()
+ }
+ joinThread(completion)
+ joinThread(holder)
+ val error = completionError.get
+ assert(error.isInstanceOf[TaskCompletionListenerException], error)
+ val reported = error.getSuppressed.toSeq
+ // The failure of the pre-completion listener, which came first, then the
failure to warn.
+ assert(reported.head eq preCompletionFailure)
+ assert(reported.tail.exists(isCausedBy(_, logFailure)))
+ }
+
+ test("SPARK-60222: the handover wait does not warn about a holder that has
handed over") {
+ val context = TaskContext.empty()
+ context.listenerHandoverWarnIntervalMs = 50
+ val holding = new CountDownLatch(1)
+ val release = new CountDownLatch(1)
+ context.addTaskFailureListener(new TaskFailureListener {
+ override def onTaskFailure(context: TaskContext, error: Throwable): Unit
= {
+ holding.countDown()
+ release.await(1, TimeUnit.MINUTES)
+ }
+ })
+ // While the first warning takes its stack trace, the holder hands over
and goes on: here it
+ // even exits, so its stack trace no longer shows the listeners.
+ val holder = new DaemonThread("holder", context.markTaskFailed(new
Exception("failed"))) {
+ override def getStackTrace: Array[StackTraceElement] = {
+ release.countDown()
+ join(TimeUnit.MINUTES.toMillis(1))
+ super.getStackTrace
+ }
+ }
+ holder.start()
+ assert(holding.await(1, TimeUnit.MINUTES))
+ val logAppender = new LogAppender("listener handover wait")
+ withLogAppender(logAppender, Seq(classOf[TaskContextImpl].getName),
Some(Level.WARN)) {
+
joinThread(startDaemonThread("completion")(context.markTaskCompleted(None)))
+ }
+ joinThread(holder)
+ assert(!logAppender.loggingEvents.exists(_.getLevel == Level.WARN))
+ }
+
+ test("SPARK-60222: listeners added after completion are not run by a thread
invoking other " +
+ "listeners") {
+ val context = TaskContext.empty()
+ val error = new Exception("failed")
+ context.markTaskFailed(error)
+ context.markTaskCompleted(Some(error))
+ val holding = new CountDownLatch(1)
+ val release = new CountDownLatch(1)
+ context.addPostStatusUpdateListener(new PostStatusUpdateListener {
+ override def onStatusUpdateSent(context: TaskContext): Unit = {
+ holding.countDown()
+ release.await(1, TimeUnit.MINUTES)
+ }
+ })
+ // Like the task thread once the executor has released the task's block
locks and memory.
+ val invoker =
startDaemonThread("status-update")(context.invokePostStatusUpdateListeners())
+ assert(holding.await(1, TimeUnit.MINUTES))
+ val listenerThreads = new ConcurrentLinkedQueue[(String, Thread)]()
+ try {
+ // Find the invocation taken, so they do not run the new listeners.
+ context.addTaskCompletionListener(loggingListener(listenerThreads,
"added"))
+ context.addTaskFailureListener(new TaskFailureListener {
+ override def onTaskFailure(context: TaskContext, error: Throwable):
Unit =
+ listenerThreads.add(("failure", Thread.currentThread()))
+ })
+ } finally {
+ release.countDown()
+ }
+ joinThread(invoker)
+ assert(listenerThreads.isEmpty)
+ // A listener added once no thread is invoking listeners runs it, on the
adding thread.
+ context.addTaskCompletionListener(loggingListener(listenerThreads,
"later"))
+ val current = Thread.currentThread()
+ assert(listenerThreads.asScala.toSeq === Seq(("later", current), ("added",
current)))
+ }
+
test("SPARK-46480: Add isFailed in TaskContext") {
val context = TaskContext.empty()
var isFailed = false
diff --git
a/core/src/test/scala/org/apache/spark/shuffle/streaming/StreamingShuffleSuite.scala
b/core/src/test/scala/org/apache/spark/shuffle/streaming/StreamingShuffleSuite.scala
index 578af94baa68..95571466e5ab 100644
---
a/core/src/test/scala/org/apache/spark/shuffle/streaming/StreamingShuffleSuite.scala
+++
b/core/src/test/scala/org/apache/spark/shuffle/streaming/StreamingShuffleSuite.scala
@@ -1372,68 +1372,91 @@ class StreamingShuffleSuite
}
}
- // Demonstrates why background threads must not call
context.markTaskFailed() directly:
- // doing so can cause the task thread's markTaskCompleted() to silently skip
completion
- // listeners. In production, cleanupResources() is a completion listener, so
skipping it
- // leaks threads. The reader avoids this by routing background errors
through a
- // CompletableFuture so markTaskFailed/markTaskCompleted are both called
from the task thread.
+ // Shows why background threads should not call context.markTaskFailed()
directly: the
+ // failure listeners then run on that thread, and the task thread's
markTaskCompleted() blocks
+ // until they finish before it runs the completion listeners. In production,
cleanupResources()
+ // is a completion listener, so it waits for whatever the background
thread's failure listeners
+ // wait for. The reader avoids this by routing background errors through an
ErrorNotifier so
+ // that markTaskFailed/markTaskCompleted are both called from the task
thread.
test("background markTaskFailed races with task thread markTaskCompleted") {
val readerContext = createTaskContext(sparkConf, partitionId = 0)
- TaskContext.setTaskContext(readerContext)
- // Latches to force the exact interleaving that triggers the race.
+ // Latches to force the interleaving.
val failureListenerEntered = new CountDownLatch(1)
val failureListenerProceed = new CountDownLatch(1)
- @volatile var completionListenerRan = false
+ @volatile var completionListenerThread: Thread = null
- // Add a failure listener that blocks, holding the
listenerInvocationThread guard.
+ // Add a failure listener that blocks, holding the listener invocation.
readerContext.addTaskFailureListener(new
org.apache.spark.util.TaskFailureListener {
override def onTaskFailure(context: TaskContext, error: Throwable): Unit
= {
failureListenerEntered.countDown()
- failureListenerProceed.await()
+ failureListenerProceed.await(1, TimeUnit.MINUTES)
}
})
// Add a completion listener -- stands in for cleanupResources() in
production.
readerContext.addTaskCompletionListener[Unit] { _ =>
- completionListenerRan = true
+ completionListenerThread = Thread.currentThread()
}
val error = new RuntimeException("background discovery failure")
- // Background thread calls markTaskFailed -> enters invokeListeners ->
acquires
- // listenerInvocationThread guard -> blocks in the failure listener above.
+ // Background thread calls markTaskFailed -> runs the failure listeners ->
holds the listener
+ // invocation while it blocks in the failure listener above.
+ @volatile var bgThreadError: Throwable = null
val bgThread = new Thread(() => {
TaskContext.setTaskContext(readerContext)
- readerContext.markTaskFailed(error)
+ try {
+ readerContext.markTaskFailed(error)
+ } catch {
+ case t: Throwable => bgThreadError = t
+ }
})
+ bgThread.setDaemon(true)
bgThread.start()
- // Wait for the failure listener to start (background thread now holds the
guard).
- failureListenerEntered.await()
+ // Wait for the failure listener to start (background thread now holds the
invocation).
+ assert(failureListenerEntered.await(1, TimeUnit.MINUTES))
- // Task thread calls markTaskCompleted while the guard is held.
- // invokeListeners sees listenerInvocationThread is set -> returns
immediately,
- // permanently skipping the completion listener.
- readerContext.markTaskCompleted(Some(error))
-
- // Release the failure listener so bgThread can finish.
- failureListenerProceed.countDown()
- bgThread.join()
-
- // This documents the race: the completion listener was skipped. The
reader avoids it by
- // never calling markTaskFailed from a background thread.
- assert(!completionListenerRan,
- "Expected completion listener to be skipped due to the race condition. "
+
- "If this starts passing, TaskContextImpl.invokeListeners may have been
fixed " +
- "and this test can be updated.")
-
- TaskContext.unset()
+ // Stands in for the task thread: it calls markTaskCompleted while the
invocation is held,
+ // and waits until the background thread hands the invocation over to it.
+ @volatile var taskThreadError: Throwable = null
+ val taskThread = new Thread(() => {
+ TaskContext.setTaskContext(readerContext)
+ try {
+ readerContext.markTaskCompleted(Some(error))
+ } catch {
+ case t: Throwable => taskThreadError = t
+ }
+ })
+ taskThread.setDaemon(true)
+ taskThread.start()
+ try {
+ eventually(Timeout(1.minute)) {
+ assert(!taskThread.isAlive || taskThread.getState ==
Thread.State.TIMED_WAITING)
+ }
+ assert(taskThread.isAlive, "markTaskCompleted returned instead of
waiting")
+ // The completion listener does not run while the failure listener
blocks.
+ assert(completionListenerThread == null)
+ } finally {
+ // Release the failure listener, so that the background thread hands the
invocation over.
+ failureListenerProceed.countDown()
+ }
+ taskThread.join(TimeUnit.MINUTES.toMillis(1))
+ assert(!taskThread.isAlive)
+ bgThread.join(TimeUnit.MINUTES.toMillis(1))
+ assert(!bgThread.isAlive)
+
+ // The completion listener ran on the thread that called markTaskCompleted.
+ assert(taskThreadError == null)
+ assert(bgThreadError == null)
+ assert(completionListenerThread eq taskThread)
}
- // With the CompletableFuture approach, the reader's cleanupResources runs
because
- // markTaskFailed is called from the task thread (no race with the
completion listener).
+ // With the ErrorNotifier, the reader's cleanupResources runs on the task
thread because
+ // markTaskFailed is called from the task thread too, so task completion
does not wait for a
+ // background thread running the failure listeners.
test("reader resources are cleaned up when client creation fails") {
class FailingClientReader[K, C](
handle: ShuffleHandle,
@@ -1472,8 +1495,8 @@ class StreamingShuffleSuite
// Simulate the task framework (runTaskWithListeners): markTaskFailed
then
// markTaskCompleted, both on the task thread. With the fix, the
background
- // thread does NOT call markTaskFailed, so there's no contention on the
- // listenerInvocationThread guard, and completion listeners run normally.
+ // thread does NOT call markTaskFailed, so no other thread holds the
listener
+ // invocation, and the completion listeners run without waiting.
readerContext.markTaskFailed(error)
readerContext.markTaskCompleted(Some(error))
@@ -1485,9 +1508,9 @@ class StreamingShuffleSuite
// On the writer side: when a background Netty thread records an error via
// errorNotifier.markError (instead of calling context.markTaskFailed
directly), the
// writer's write() method surfaces the error on the task thread. Because
markTaskFailed
- // and markTaskCompleted are both called from the task thread, there is no
contention on
- // the listenerInvocationThread guard in TaskContextImpl, and the completion
listener
- // (cleanupResources) runs normally.
+ // and markTaskCompleted are both called from the task thread, no other
thread holds the
+ // listener invocation in TaskContextImpl, and the completion listener
(cleanupResources)
+ // runs without waiting.
test("writer resources are cleaned up when server handler " +
"receive fails via errorNotifier") {
withSpark(new SparkContext("local", "StreamingShuffleSuite", sparkConf)) {
sc =>
@@ -1546,9 +1569,9 @@ class StreamingShuffleSuite
// Simulate the task framework (runTaskWithListeners): markTaskFailed
then
// markTaskCompleted, both on the task thread. With the ErrorNotifier
fix, the
- // background Netty thread does NOT call markTaskFailed, so there is no
contention
- // on the listenerInvocationThread guard, and the completion listener
- // (cleanupResources) runs normally.
+ // background Netty thread does NOT call markTaskFailed, so no other
thread holds
+ // the listener invocation, and the completion listener
(cleanupResources) runs
+ // without waiting.
writerContext.markTaskFailed(writeError)
writerContext.markTaskCompleted(Some(writeError))
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]