This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.3 by this push:
     new d5aa6ee8ebb7 [SPARK-60469][SS] Fix the executor termination check in 
async progress tracking cleanup
d5aa6ee8ebb7 is described below

commit d5aa6ee8ebb7674988d15222e7496cd7257785fa
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Oct 22 14:12:35 2026 -0700

    [SPARK-60469][SS] Fix the executor termination check in async progress 
tracking cleanup
    
    ### What changes were proposed in this pull request?
    
    This PR fixes the check that 
`AsyncProgressTrackingMicroBatchExecution.cleanup()` uses to decide whether the 
async log write executor stopped in time. SPARK-58131 added this check.
    
    - Check `isTerminated` instead of `isShutdown`.
    - If the executor has not terminated after the 30-second wait, log a 
warning and continue the cleanup instead of throwing `TimeoutException`.
    
    ### Why are the changes needed?
    
    `cleanup()` stops the executor and then checks 
`asyncWritesExecutorService.isShutdown`. `isShutdown` is true as soon as 
`shutdown()` or `shutdownNow()` is called, so the `else` branch can never run.
    
    - The warning and the `TimeoutException` are dead code.
    - The INFO message `... has been shutdown` is logged even when an offset or 
commit log write is still running after the 30-second wait.
    
    The write can keep running after the wait because `ThreadUtils.shutdown` 
does not call `shutdownNow()` after the grace period. SPARK-60466 fixes that 
separately. This PR still calls `ThreadUtils.shutdown`, so it gets that fix too.
    
    Making the `TimeoutException` reachable would be worse. `cleanup()` runs in 
the `finally` block of `StreamExecution.runStream`, before the rest of the 
query termination, so an exception from `cleanup()` skips that work. I tried it 
with a commit log write blocked during `query.stop()`:
    
    - `query.stop()` still returned normally after 30 seconds, because the 
exception is thrown in the stream execution thread and goes to its uncaught 
exception handler.
    - The query stayed in `spark.streams.active`, and no `QueryTerminatedEvent` 
was posted.
    - Restarting the query from the same checkpoint failed with 
`[CONCURRENT_QUERY] Another instance of this query was just started by a 
concurrent session.`
    
    ### Does this PR introduce _any_ user-facing change?
    
    Only a log message changes. When async log writes are still running 30 
seconds after a query stops, Spark now logs the WARN message `... did not 
terminate in 30 seconds` instead of the INFO message `... has been shutdown`. 
Query stop behaves as before.
    
    SPARK-58131 is not in any released version yet. It is in `master`, 
`branch-4.x` and `branch-4.3`, and in the 
[v4.3.0-rc1](https://github.com/apache/spark/releases/tag/v4.3.0-rc1) 
(2026-08-31) and 
[v4.3.0-rc2](https://github.com/apache/spark/releases/tag/v4.3.0-rc2) 
(2026-10-12) tags.
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test case. The test blocks the async 
commit log write of a batch, stops the query, and then checks the log message 
and that the query is no longer active. Without this fix, the test fails 
because only the INFO message is logged.
    
    The test takes about 30 seconds because the wait in `cleanup()` is fixed at 
30 seconds.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #59710 from dongjoon-hyun/SPARK-60469.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit 10a34a06707851ce0ab0afebee3d6c19cb757f27)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../AsyncProgressTrackingMicroBatchExecution.scala |  8 +--
 ...cProgressTrackingMicroBatchExecutionSuite.scala | 59 +++++++++++++++++++++-
 2 files changed, 62 insertions(+), 5 deletions(-)

diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/AsyncProgressTrackingMicroBatchExecution.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/AsyncProgressTrackingMicroBatchExecution.scala
index a87c4c3574d1..e01f42f67817 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/AsyncProgressTrackingMicroBatchExecution.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/AsyncProgressTrackingMicroBatchExecution.scala
@@ -368,14 +368,14 @@ class AsyncProgressTrackingMicroBatchExecution(
     } else {
       ThreadUtils.shutdown(asyncWritesExecutorService)
     }
-    if (asyncWritesExecutorService.isShutdown) {
+    // Only warn if the executor has not terminated. An exception here would 
skip the rest of the
+    // query termination in StreamExecution.runStream, such as unregistering 
the query.
+    if (asyncWritesExecutorService.isTerminated) {
       logInfo(log"Async progress tracking executor for query " +
         log"${MDC(PRETTY_ID_STRING, prettyIdString)} has been shutdown")
     } else {
       logWarning(log"Async progress tracking executor for query " +
-        log"${MDC(PRETTY_ID_STRING, prettyIdString)} failed to shutdown 
properly")
-      throw new TimeoutException(
-        "Failed to shutdown Async Progress Tracking executor in 30 seconds")
+        log"${MDC(PRETTY_ID_STRING, prettyIdString)} did not terminate in 30 
seconds")
     }
     commitAllSources()
     super.cleanup()
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/AsyncProgressTrackingMicroBatchExecutionSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/AsyncProgressTrackingMicroBatchExecutionSuite.scala
index 9d5b5b1bfce4..35bb20338da8 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/AsyncProgressTrackingMicroBatchExecutionSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/AsyncProgressTrackingMicroBatchExecutionSuite.scala
@@ -22,6 +22,7 @@ import java.util.concurrent.{CountDownLatch, 
ExecutionException, Semaphore, Time
 
 import scala.collection.mutable.ListBuffer
 
+import org.apache.logging.log4j.Level
 import org.scalatest.BeforeAndAfter
 import org.scalatest.matchers.should.Matchers
 import org.scalatest.time.{Seconds, Span}
@@ -30,8 +31,9 @@ import org.apache.spark.{SparkIllegalArgumentException, 
TestUtils}
 import org.apache.spark.sql._
 import org.apache.spark.sql.connector.read.streaming
 import org.apache.spark.sql.execution.streaming.checkpointing.{AsyncCommitLog, 
AsyncOffsetSeqLog, CommitMetadata, OffsetSeq}
-import 
org.apache.spark.sql.execution.streaming.runtime.{AsyncProgressTrackingMicroBatchExecution,
 LongOffset, MemoryStream, StreamExecution}
+import 
org.apache.spark.sql.execution.streaming.runtime.{AsyncProgressTrackingMicroBatchExecution,
 LongOffset, MemoryStream, StreamExecution, StreamingQueryWrapper}
 import 
org.apache.spark.sql.execution.streaming.runtime.AsyncProgressTrackingMicroBatchExecution.{ASYNC_PROGRESS_TRACKING_CHECKPOINTING_INTERVAL_MS,
 ASYNC_PROGRESS_TRACKING_ENABLED, 
ASYNC_PROGRESS_TRACKING_OVERRIDE_SINK_SUPPORT_CHECK}
+import 
org.apache.spark.sql.execution.streaming.state.{FailureInjectionCheckpointFileManager,
 FailureInjectionFileSystem}
 import org.apache.spark.sql.functions.{column, window}
 import org.apache.spark.sql.internal.SQLConf
 import org.apache.spark.sql.streaming.{StreamingQuery, 
StreamingQueryException, StreamTest, Trigger}
@@ -866,6 +868,61 @@ class AsyncProgressTrackingMicroBatchExecutionSuite
     }
   }
 
+  test("SPARK-60469: stop warns and still terminates when async log writes do 
not finish") {
+    withSQLConf(
+      SQLConf.STREAMING_CHECKPOINT_FILE_MANAGER_CLASS.parent.key ->
+        classOf[FailureInjectionCheckpointFileManager].getName) {
+      withTempDir { checkpointDir =>
+        val injectionState = 
FailureInjectionFileSystem.registerTempPath(checkpointDir.getPath)
+        val semaphore = FailureInjectionFileSystem.createAtomicDelaySemaphore
+        try {
+          val inputData = MemoryStream[Int]
+          val query = inputData.toDF().writeStream
+            .format("noop")
+            .option(ASYNC_PROGRESS_TRACKING_ENABLED, true)
+            .option(ASYNC_PROGRESS_TRACKING_CHECKPOINTING_INTERVAL_MS, 0)
+            .option("checkpointLocation", checkpointDir.getAbsolutePath)
+            .start()
+          inputData.addData(0)
+          query.processAllAvailable()
+
+          // Block the async commit log write of batch 1 so that it is still 
running on stop.
+          injectionState.delayCreateAtomicRegex = Seq(".*/commits/1")
+          inputData.addData(1)
+          query.processAllAvailable()
+          eventually(timeout(streamingTimeout)) {
+            assert(semaphore.hasQueuedThreads)
+          }
+
+          val logAppender = new LogAppender("async progress tracking executor 
shutdown")
+          logAppender.setThreshold(Level.WARN)
+          try {
+            withLogAppender(logAppender) {
+              // This waits 30 seconds for the blocked write.
+              query.stop()
+            }
+          } finally {
+            // Let the blocked write finish before the checkpoint directory is 
deleted, and drop
+            // the permit if the write was interrupted instead.
+            injectionState.delayCreateAtomicRegex = Seq.empty
+            semaphore.release()
+            
waitPendingOffsetWrites(query.asInstanceOf[StreamingQueryWrapper].streamingQuery)
+            semaphore.drainPermits()
+          }
+          
assert(logAppender.loggingEvents.exists(_.getMessage.getFormattedMessage.endsWith(
+            "did not terminate in 30 seconds")))
+          assert(!spark.streams.active.exists(_.id == query.id))
+        } finally {
+          injectionState.delayCreateAtomicRegex = Seq.empty
+          if (semaphore.hasQueuedThreads) {
+            semaphore.release()
+          }
+          
FailureInjectionFileSystem.removePathFromTempToInjectionState(checkpointDir.getPath)
+        }
+      }
+    }
+  }
+
   test("commit intervals happy path") {
 
     val checkpointLocation = Utils.createTempDir(namePrefix = 
"streaming.metadata").getCanonicalPath


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to