This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 1379fdd51dfe [SPARK-60992][CORE] Fix marking a succeeded task as
failed when a post-status-update listener fails
1379fdd51dfe is described below
commit 1379fdd51dfedf6a0f87aa2116354262daa78952
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Nov 6 08:20:21 2026 -0800
[SPARK-60992][CORE] Fix marking a succeeded task as failed when a
post-status-update listener fails
### What changes were proposed in this pull request?
`TaskContextImpl.invokePostStatusUpdateListeners()` no longer marks the
task failed when a post-status-update listener throws. The exceptions are
collected, all listeners still run, and a `TaskCompletionListenerException` is
thrown afterwards so that `Executor` keeps logging it through
`Utils.tryLogNonFatalError`, as before.
The scaladoc of `TaskContext.addPostStatusUpdateListener` now says that
listener failures are only logged.
### Why are the changes needed?
`Executor.TaskRunner.run` invokes the post-status-update listeners (added
by SPARK-57491 for the push-based shuffle block push) after it has sent
`TaskState.FINISHED` to the driver. At that point `Task.run` has unset the
`TaskContext`, and the executor has released the task's block locks and memory.
`invokePostStatusUpdateListeners()` reuses `invokeListeners()`, which calls
`markTaskFailed` for any listener that throws. So a failing post-status-update
listener makes a task that was already reported as successful look failed:
- `isFailed()` turns true and `getTaskFailure` is defined.
- The task failure listeners run on the task thread, without a
`TaskContext` and after the task's memory is released. In the repro,
`TaskContext.get()` is `null` and the task memory is 0 inside the failure
listener.
The driver still has the successful result, so this is only visible to code
that reads the task state or registers failure listeners (for example,
`RocksDBStateStoreProvider` aborts a state store on failure, and a read-only
store that was already released logs a misleading
`STATE_STORE_OPERATION_OUT_OF_ORDER` error).
The only in-tree post-status-update listener is the push-based shuffle one,
so this is unlikely to be hit today. It is still wrong for the task to change
state after the status update was sent.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Added a test to `TaskContextSuite`. It fails without the change (a failure
listener is invoked and `isFailed()` is true) and passes with it.
```
build/sbt 'core/testOnly *TaskContextSuite *ExecutorSuite
*ShuffleBlockPusherSuite'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #60164 from dongjoon-hyun/SPARK-60992.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 6f9123594fc7df866d28110bc9d68b2d91f592b7)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../main/scala/org/apache/spark/TaskContext.scala | 3 +
.../scala/org/apache/spark/TaskContextImpl.scala | 33 ++++++++-
.../apache/spark/scheduler/TaskContextSuite.scala | 86 ++++++++++++++++++++++
3 files changed, 118 insertions(+), 4 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/TaskContext.scala
b/core/src/main/scala/org/apache/spark/TaskContext.scala
index 34df7d5c6ccc..83fe99c230c7 100644
--- a/core/src/main/scala/org/apache/spark/TaskContext.scala
+++ b/core/src/main/scala/org/apache/spark/TaskContext.scala
@@ -220,6 +220,9 @@ abstract class TaskContext extends Serializable {
* avoiding stale data being merged without detection.
*
* The callback runs on the same executor thread that sends the status
update.
+ *
+ * Exceptions thrown by the listener are only logged by the executor. They
do not mark the task
+ * as failed, since the driver has already been told that it succeeded.
*/
private[spark] def addPostStatusUpdateListener(listener:
PostStatusUpdateListener): TaskContext
diff --git a/core/src/main/scala/org/apache/spark/TaskContextImpl.scala
b/core/src/main/scala/org/apache/spark/TaskContextImpl.scala
index 45e8808812b7..ab9e6800c186 100644
--- a/core/src/main/scala/org/apache/spark/TaskContextImpl.scala
+++ b/core/src/main/scala/org/apache/spark/TaskContextImpl.scala
@@ -207,8 +207,31 @@ private[spark] class TaskContextImpl(
}
private[spark] def invokePostStatusUpdateListeners(): Unit = {
+ // The executor has already reported the task's success to the driver and
released the task's
+ // block locks and memory, and `Task.run` has unset its `TaskContext`. So
a failing listener
+ // must not mark the task failed, which `invokeListeners()` does for a
listener that throws:
+ // the failure listeners would run for a task that has not failed, without
those resources.
+ val listenerExceptions = new ArrayBuffer[Throwable](2)
invokeListeners(onPostStatusUpdateCallbacks, "PostStatusUpdateListener",
None, Claim) {
- _.onStatusUpdateSent(this)
+ listener =>
+ try {
+ listener.onStatusUpdateSent(this)
+ } catch {
+ case e: Throwable =>
+ listenerExceptions += e
+ // Log each failure with its own exception, as `invokeListeners()`
does: a structured
+ // log keeps only the message of the aggregate's suppressed
exceptions. Best effort:
+ // a failure to log must not reach `invokeListeners()`, which
would mark the task
+ // failed. It is reported with `e` instead.
+ try {
+ logError(log"Error in ${MDC(LISTENER,
"PostStatusUpdateListener")}", e)
+ } catch {
+ case t: Throwable => e.addSuppressed(t)
+ }
+ }
+ }
+ if (listenerExceptions.nonEmpty) {
+ throw newTaskCompletionListenerException(listenerExceptions.toSeq, None)
}
}
@@ -561,9 +584,11 @@ private[spark] class TaskContextImpl(
* left to it, since its thread may be one on which they must not run, e.g.
a kill thread, or the
* task thread once `Task.run` has unset its `TaskContext` and the executor
has released the
* task's block locks and memory. A listener that fails here still starts
the failure listeners
- * on this thread (see `runListener`). Unless `mode` is `Nested`, this
thread releases the
- * invocation when it is done, or hands it over to a thread waiting for it in
- * `markTaskCompleted()`.
+ * on this thread (see `runListener`), except for the listeners of
+ * `invokePostStatusUpdateListeners()`, which collect their failures
themselves, so that they
+ * neither mark the task failed nor start the failure listeners. Unless
`mode` is `Nested`, this
+ * thread releases the invocation when it is done, or hands it over to a
thread waiting for it
+ * in `markTaskCompleted()`.
*
* Exceptions of the failure listeners left to this invocation are reported
to the caller of
* this method, together with those of `listeners`.
diff --git
a/core/src/test/scala/org/apache/spark/scheduler/TaskContextSuite.scala
b/core/src/test/scala/org/apache/spark/scheduler/TaskContextSuite.scala
index f1454ac03b5c..dfb78afd4171 100644
--- a/core/src/test/scala/org/apache/spark/scheduler/TaskContextSuite.scala
+++ b/core/src/test/scala/org/apache/spark/scheduler/TaskContextSuite.scala
@@ -1988,6 +1988,92 @@ class TaskContextSuite extends SparkFunSuite with
BeforeAndAfter with LocalSpark
assert(!completionInvoked) // completion listener should NOT be triggered
}
+ test("SPARK-60992: a failing post-status-update listener does not mark the
task failed") {
+ val context = TaskContext.empty()
+ val failureListener = mock(classOf[TaskFailureListener])
+ context.addTaskFailureListener(failureListener)
+ context.markTaskCompleted(None)
+
+ var isFailedInLastListener = true
+ context.addPostStatusUpdateListener(new PostStatusUpdateListener {
+ override def onStatusUpdateSent(context: TaskContext): Unit =
+ isFailedInLastListener = context.isFailed()
+ })
+ val error = new RuntimeException("post-status-update failure")
+ val anotherError = new RuntimeException("another post-status-update
failure")
+ context.addPostStatusUpdateListener(new PostStatusUpdateListener {
+ override def onStatusUpdateSent(context: TaskContext): Unit = throw error
+ })
+ context.addPostStatusUpdateListener(new PostStatusUpdateListener {
+ override def onStatusUpdateSent(context: TaskContext): Unit = throw
anotherError
+ })
+
+ // The caller, the executor, still gets all failures to log them, in
invocation order.
+ val e = intercept[TaskCompletionListenerException] {
+ context.invokePostStatusUpdateListeners()
+ }
+ assert(e.getMessage.contains("post-status-update failure"))
+ assert(e.getSuppressed.toSeq === Seq(anotherError, error))
+
+ // The executor has already reported the task's success and released its
resources.
+ verify(failureListener, never()).onTaskFailure(any(), any())
+ assert(!isFailedInLastListener)
+ assert(!context.isFailed())
+ assert(context.getTaskFailure.isEmpty)
+
+ // The task is still not failed for a failure listener added later.
+ val lateFailureListener = mock(classOf[TaskFailureListener])
+ context.addTaskFailureListener(lateFailureListener)
+ verify(lateFailureListener, never()).onTaskFailure(any(), any())
+ }
+
+ test("SPARK-60992: each failing post-status-update listener is logged with
its exception") {
+ val context = TaskContext.empty()
+ val first = new RuntimeException("first failure")
+ val second = new RuntimeException("second failure")
+ context.addPostStatusUpdateListener(new PostStatusUpdateListener {
+ override def onStatusUpdateSent(context: TaskContext): Unit = throw first
+ })
+ context.addPostStatusUpdateListener(new PostStatusUpdateListener {
+ override def onStatusUpdateSent(context: TaskContext): Unit = throw
second
+ })
+
+ val appender = new LogAppender("post-status-update listener failures")
+ withLogAppender(appender, Seq(classOf[TaskContextImpl].getName),
Some(Level.ERROR)) {
+ intercept[TaskCompletionListenerException] {
+ context.invokePostStatusUpdateListeners()
+ }
+ }
+ val logged = appender.loggingEvents.filter(_.getMessage.getFormattedMessage
+ .contains("Error in PostStatusUpdateListener"))
+ assert(logged.map(_.getThrown).toSeq === Seq(second, first))
+ }
+
+ test("SPARK-60992: a failure to log a post-status-update listener's failure
does not mark " +
+ "the task failed") {
+ val context = TaskContext.empty()
+ val failureListener = mock(classOf[TaskFailureListener])
+ context.addTaskFailureListener(failureListener)
+ val error = new RuntimeException("post-status-update failure")
+ context.addPostStatusUpdateListener(new PostStatusUpdateListener {
+ override def onStatusUpdateSent(context: TaskContext): Unit = throw error
+ })
+
+ val logFailure = new IllegalStateException("log failure")
+ val failingAppender = new FailingAppender("Error in
PostStatusUpdateListener", logFailure)
+ withLogAppender(failingAppender, Seq(classOf[TaskContextImpl].getName),
Some(Level.ERROR)) {
+ val e = intercept[TaskCompletionListenerException] {
+ context.invokePostStatusUpdateListeners()
+ }
+ assert(e.getSuppressed.toSeq === Seq(error))
+ // The failure to log is reported with the failure it was logging.
+ val causes =
Iterator.iterate[Throwable](error.getSuppressed.head)(_.getCause)
+ assert(causes.takeWhile(_ != null).contains(logFailure))
+ }
+ verify(failureListener, never()).onTaskFailure(any(), any())
+ assert(!context.isFailed())
+ }
+
test("SPARK-46947: ensure the correct block manager is used to unroll memory
for task") {
import BlockManagerValidationPlugin._
BlockManagerValidationPlugin.resetState()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]