This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 1379fdd51dfe [SPARK-60992][CORE] Fix marking a succeeded task as 
failed when a post-status-update listener fails
1379fdd51dfe is described below

commit 1379fdd51dfedf6a0f87aa2116354262daa78952
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Nov 6 08:20:21 2026 -0800

    [SPARK-60992][CORE] Fix marking a succeeded task as failed when a 
post-status-update listener fails
    
    ### What changes were proposed in this pull request?
    
    `TaskContextImpl.invokePostStatusUpdateListeners()` no longer marks the 
task failed when a post-status-update listener throws. The exceptions are 
collected, all listeners still run, and a `TaskCompletionListenerException` is 
thrown afterwards so that `Executor` keeps logging it through 
`Utils.tryLogNonFatalError`, as before.
    
    The scaladoc of `TaskContext.addPostStatusUpdateListener` now says that 
listener failures are only logged.
    
    ### Why are the changes needed?
    
    `Executor.TaskRunner.run` invokes the post-status-update listeners (added 
by SPARK-57491 for the push-based shuffle block push) after it has sent 
`TaskState.FINISHED` to the driver. At that point `Task.run` has unset the 
`TaskContext`, and the executor has released the task's block locks and memory.
    
    `invokePostStatusUpdateListeners()` reuses `invokeListeners()`, which calls 
`markTaskFailed` for any listener that throws. So a failing post-status-update 
listener makes a task that was already reported as successful look failed:
    - `isFailed()` turns true and `getTaskFailure` is defined.
    - The task failure listeners run on the task thread, without a 
`TaskContext` and after the task's memory is released. In the repro, 
`TaskContext.get()` is `null` and the task memory is 0 inside the failure 
listener.
    
    The driver still has the successful result, so this is only visible to code 
that reads the task state or registers failure listeners (for example, 
`RocksDBStateStoreProvider` aborts a state store on failure, and a read-only 
store that was already released logs a misleading 
`STATE_STORE_OPERATION_OUT_OF_ORDER` error).
    
    The only in-tree post-status-update listener is the push-based shuffle one, 
so this is unlikely to be hit today. It is still wrong for the task to change 
state after the status update was sent.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    Added a test to `TaskContextSuite`. It fails without the change (a failure 
listener is invoked and `isFailed()` is true) and passes with it.
    
    ```
    build/sbt 'core/testOnly *TaskContextSuite *ExecutorSuite 
*ShuffleBlockPusherSuite'
    ```
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #60164 from dongjoon-hyun/SPARK-60992.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit 6f9123594fc7df866d28110bc9d68b2d91f592b7)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../main/scala/org/apache/spark/TaskContext.scala  |  3 +
 .../scala/org/apache/spark/TaskContextImpl.scala   | 33 ++++++++-
 .../apache/spark/scheduler/TaskContextSuite.scala  | 86 ++++++++++++++++++++++
 3 files changed, 118 insertions(+), 4 deletions(-)

diff --git a/core/src/main/scala/org/apache/spark/TaskContext.scala 
b/core/src/main/scala/org/apache/spark/TaskContext.scala
index 34df7d5c6ccc..83fe99c230c7 100644
--- a/core/src/main/scala/org/apache/spark/TaskContext.scala
+++ b/core/src/main/scala/org/apache/spark/TaskContext.scala
@@ -220,6 +220,9 @@ abstract class TaskContext extends Serializable {
    * avoiding stale data being merged without detection.
    *
    * The callback runs on the same executor thread that sends the status 
update.
+   *
+   * Exceptions thrown by the listener are only logged by the executor. They 
do not mark the task
+   * as failed, since the driver has already been told that it succeeded.
    */
   private[spark] def addPostStatusUpdateListener(listener: 
PostStatusUpdateListener): TaskContext
 
diff --git a/core/src/main/scala/org/apache/spark/TaskContextImpl.scala 
b/core/src/main/scala/org/apache/spark/TaskContextImpl.scala
index 45e8808812b7..ab9e6800c186 100644
--- a/core/src/main/scala/org/apache/spark/TaskContextImpl.scala
+++ b/core/src/main/scala/org/apache/spark/TaskContextImpl.scala
@@ -207,8 +207,31 @@ private[spark] class TaskContextImpl(
   }
 
   private[spark] def invokePostStatusUpdateListeners(): Unit = {
+    // The executor has already reported the task's success to the driver and 
released the task's
+    // block locks and memory, and `Task.run` has unset its `TaskContext`. So 
a failing listener
+    // must not mark the task failed, which `invokeListeners()` does for a 
listener that throws:
+    // the failure listeners would run for a task that has not failed, without 
those resources.
+    val listenerExceptions = new ArrayBuffer[Throwable](2)
     invokeListeners(onPostStatusUpdateCallbacks, "PostStatusUpdateListener", 
None, Claim) {
-      _.onStatusUpdateSent(this)
+      listener =>
+        try {
+          listener.onStatusUpdateSent(this)
+        } catch {
+          case e: Throwable =>
+            listenerExceptions += e
+            // Log each failure with its own exception, as `invokeListeners()` 
does: a structured
+            // log keeps only the message of the aggregate's suppressed 
exceptions. Best effort:
+            // a failure to log must not reach `invokeListeners()`, which 
would mark the task
+            // failed. It is reported with `e` instead.
+            try {
+              logError(log"Error in ${MDC(LISTENER, 
"PostStatusUpdateListener")}", e)
+            } catch {
+              case t: Throwable => e.addSuppressed(t)
+            }
+        }
+    }
+    if (listenerExceptions.nonEmpty) {
+      throw newTaskCompletionListenerException(listenerExceptions.toSeq, None)
     }
   }
 
@@ -561,9 +584,11 @@ private[spark] class TaskContextImpl(
    * left to it, since its thread may be one on which they must not run, e.g. 
a kill thread, or the
    * task thread once `Task.run` has unset its `TaskContext` and the executor 
has released the
    * task's block locks and memory. A listener that fails here still starts 
the failure listeners
-   * on this thread (see `runListener`). Unless `mode` is `Nested`, this 
thread releases the
-   * invocation when it is done, or hands it over to a thread waiting for it in
-   * `markTaskCompleted()`.
+   * on this thread (see `runListener`), except for the listeners of
+   * `invokePostStatusUpdateListeners()`, which collect their failures 
themselves, so that they
+   * neither mark the task failed nor start the failure listeners. Unless 
`mode` is `Nested`, this
+   * thread releases the invocation when it is done, or hands it over to a 
thread waiting for it
+   * in `markTaskCompleted()`.
    *
    * Exceptions of the failure listeners left to this invocation are reported 
to the caller of
    * this method, together with those of `listeners`.
diff --git 
a/core/src/test/scala/org/apache/spark/scheduler/TaskContextSuite.scala 
b/core/src/test/scala/org/apache/spark/scheduler/TaskContextSuite.scala
index f1454ac03b5c..dfb78afd4171 100644
--- a/core/src/test/scala/org/apache/spark/scheduler/TaskContextSuite.scala
+++ b/core/src/test/scala/org/apache/spark/scheduler/TaskContextSuite.scala
@@ -1988,6 +1988,92 @@ class TaskContextSuite extends SparkFunSuite with 
BeforeAndAfter with LocalSpark
     assert(!completionInvoked) // completion listener should NOT be triggered
   }
 
+  test("SPARK-60992: a failing post-status-update listener does not mark the 
task failed") {
+    val context = TaskContext.empty()
+    val failureListener = mock(classOf[TaskFailureListener])
+    context.addTaskFailureListener(failureListener)
+    context.markTaskCompleted(None)
+
+    var isFailedInLastListener = true
+    context.addPostStatusUpdateListener(new PostStatusUpdateListener {
+      override def onStatusUpdateSent(context: TaskContext): Unit =
+        isFailedInLastListener = context.isFailed()
+    })
+    val error = new RuntimeException("post-status-update failure")
+    val anotherError = new RuntimeException("another post-status-update 
failure")
+    context.addPostStatusUpdateListener(new PostStatusUpdateListener {
+      override def onStatusUpdateSent(context: TaskContext): Unit = throw error
+    })
+    context.addPostStatusUpdateListener(new PostStatusUpdateListener {
+      override def onStatusUpdateSent(context: TaskContext): Unit = throw 
anotherError
+    })
+
+    // The caller, the executor, still gets all failures to log them, in 
invocation order.
+    val e = intercept[TaskCompletionListenerException] {
+      context.invokePostStatusUpdateListeners()
+    }
+    assert(e.getMessage.contains("post-status-update failure"))
+    assert(e.getSuppressed.toSeq === Seq(anotherError, error))
+
+    // The executor has already reported the task's success and released its 
resources.
+    verify(failureListener, never()).onTaskFailure(any(), any())
+    assert(!isFailedInLastListener)
+    assert(!context.isFailed())
+    assert(context.getTaskFailure.isEmpty)
+
+    // The task is still not failed for a failure listener added later.
+    val lateFailureListener = mock(classOf[TaskFailureListener])
+    context.addTaskFailureListener(lateFailureListener)
+    verify(lateFailureListener, never()).onTaskFailure(any(), any())
+  }
+
+  test("SPARK-60992: each failing post-status-update listener is logged with 
its exception") {
+    val context = TaskContext.empty()
+    val first = new RuntimeException("first failure")
+    val second = new RuntimeException("second failure")
+    context.addPostStatusUpdateListener(new PostStatusUpdateListener {
+      override def onStatusUpdateSent(context: TaskContext): Unit = throw first
+    })
+    context.addPostStatusUpdateListener(new PostStatusUpdateListener {
+      override def onStatusUpdateSent(context: TaskContext): Unit = throw 
second
+    })
+
+    val appender = new LogAppender("post-status-update listener failures")
+    withLogAppender(appender, Seq(classOf[TaskContextImpl].getName), 
Some(Level.ERROR)) {
+      intercept[TaskCompletionListenerException] {
+        context.invokePostStatusUpdateListeners()
+      }
+    }
+    val logged = appender.loggingEvents.filter(_.getMessage.getFormattedMessage
+      .contains("Error in PostStatusUpdateListener"))
+    assert(logged.map(_.getThrown).toSeq === Seq(second, first))
+  }
+
+  test("SPARK-60992: a failure to log a post-status-update listener's failure 
does not mark " +
+      "the task failed") {
+    val context = TaskContext.empty()
+    val failureListener = mock(classOf[TaskFailureListener])
+    context.addTaskFailureListener(failureListener)
+    val error = new RuntimeException("post-status-update failure")
+    context.addPostStatusUpdateListener(new PostStatusUpdateListener {
+      override def onStatusUpdateSent(context: TaskContext): Unit = throw error
+    })
+
+    val logFailure = new IllegalStateException("log failure")
+    val failingAppender = new FailingAppender("Error in 
PostStatusUpdateListener", logFailure)
+    withLogAppender(failingAppender, Seq(classOf[TaskContextImpl].getName), 
Some(Level.ERROR)) {
+      val e = intercept[TaskCompletionListenerException] {
+        context.invokePostStatusUpdateListeners()
+      }
+      assert(e.getSuppressed.toSeq === Seq(error))
+      // The failure to log is reported with the failure it was logging.
+      val causes = 
Iterator.iterate[Throwable](error.getSuppressed.head)(_.getCause)
+      assert(causes.takeWhile(_ != null).contains(logFailure))
+    }
+    verify(failureListener, never()).onTaskFailure(any(), any())
+    assert(!context.isFailed())
+  }
+
   test("SPARK-46947: ensure the correct block manager is used to unroll memory 
for task") {
     import BlockManagerValidationPlugin._
     BlockManagerValidationPlugin.resetState()


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to