This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.3 by this push:
new d5aa6ee8ebb7 [SPARK-60469][SS] Fix the executor termination check in
async progress tracking cleanup
d5aa6ee8ebb7 is described below
commit d5aa6ee8ebb7674988d15222e7496cd7257785fa
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Oct 22 14:12:35 2026 -0700
[SPARK-60469][SS] Fix the executor termination check in async progress
tracking cleanup
### What changes were proposed in this pull request?
This PR fixes the check that
`AsyncProgressTrackingMicroBatchExecution.cleanup()` uses to decide whether the
async log write executor stopped in time. SPARK-58131 added this check.
- Check `isTerminated` instead of `isShutdown`.
- If the executor has not terminated after the 30-second wait, log a
warning and continue the cleanup instead of throwing `TimeoutException`.
### Why are the changes needed?
`cleanup()` stops the executor and then checks
`asyncWritesExecutorService.isShutdown`. `isShutdown` is true as soon as
`shutdown()` or `shutdownNow()` is called, so the `else` branch can never run.
- The warning and the `TimeoutException` are dead code.
- The INFO message `... has been shutdown` is logged even when an offset or
commit log write is still running after the 30-second wait.
The write can keep running after the wait because `ThreadUtils.shutdown`
does not call `shutdownNow()` after the grace period. SPARK-60466 fixes that
separately. This PR still calls `ThreadUtils.shutdown`, so it gets that fix too.
Making the `TimeoutException` reachable would be worse. `cleanup()` runs in
the `finally` block of `StreamExecution.runStream`, before the rest of the
query termination, so an exception from `cleanup()` skips that work. I tried it
with a commit log write blocked during `query.stop()`:
- `query.stop()` still returned normally after 30 seconds, because the
exception is thrown in the stream execution thread and goes to its uncaught
exception handler.
- The query stayed in `spark.streams.active`, and no `QueryTerminatedEvent`
was posted.
- Restarting the query from the same checkpoint failed with
`[CONCURRENT_QUERY] Another instance of this query was just started by a
concurrent session.`
### Does this PR introduce _any_ user-facing change?
Only a log message changes. When async log writes are still running 30
seconds after a query stops, Spark now logs the WARN message `... did not
terminate in 30 seconds` instead of the INFO message `... has been shutdown`.
Query stop behaves as before.
SPARK-58131 is not in any released version yet. It is in `master`,
`branch-4.x` and `branch-4.3`, and in the
[v4.3.0-rc1](https://github.com/apache/spark/releases/tag/v4.3.0-rc1)
(2026-08-31) and
[v4.3.0-rc2](https://github.com/apache/spark/releases/tag/v4.3.0-rc2)
(2026-10-12) tags.
### How was this patch tested?
Pass the CIs with the newly added test case. The test blocks the async
commit log write of a batch, stops the query, and then checks the log message
and that the query is no longer active. Without this fix, the test fails
because only the INFO message is logged.
The test takes about 30 seconds because the wait in `cleanup()` is fixed at
30 seconds.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #59710 from dongjoon-hyun/SPARK-60469.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 10a34a06707851ce0ab0afebee3d6c19cb757f27)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../AsyncProgressTrackingMicroBatchExecution.scala | 8 +--
...cProgressTrackingMicroBatchExecutionSuite.scala | 59 +++++++++++++++++++++-
2 files changed, 62 insertions(+), 5 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/AsyncProgressTrackingMicroBatchExecution.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/AsyncProgressTrackingMicroBatchExecution.scala
index a87c4c3574d1..e01f42f67817 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/AsyncProgressTrackingMicroBatchExecution.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/AsyncProgressTrackingMicroBatchExecution.scala
@@ -368,14 +368,14 @@ class AsyncProgressTrackingMicroBatchExecution(
} else {
ThreadUtils.shutdown(asyncWritesExecutorService)
}
- if (asyncWritesExecutorService.isShutdown) {
+ // Only warn if the executor has not terminated. An exception here would
skip the rest of the
+ // query termination in StreamExecution.runStream, such as unregistering
the query.
+ if (asyncWritesExecutorService.isTerminated) {
logInfo(log"Async progress tracking executor for query " +
log"${MDC(PRETTY_ID_STRING, prettyIdString)} has been shutdown")
} else {
logWarning(log"Async progress tracking executor for query " +
- log"${MDC(PRETTY_ID_STRING, prettyIdString)} failed to shutdown
properly")
- throw new TimeoutException(
- "Failed to shutdown Async Progress Tracking executor in 30 seconds")
+ log"${MDC(PRETTY_ID_STRING, prettyIdString)} did not terminate in 30
seconds")
}
commitAllSources()
super.cleanup()
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/AsyncProgressTrackingMicroBatchExecutionSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/AsyncProgressTrackingMicroBatchExecutionSuite.scala
index 9d5b5b1bfce4..35bb20338da8 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/AsyncProgressTrackingMicroBatchExecutionSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/AsyncProgressTrackingMicroBatchExecutionSuite.scala
@@ -22,6 +22,7 @@ import java.util.concurrent.{CountDownLatch,
ExecutionException, Semaphore, Time
import scala.collection.mutable.ListBuffer
+import org.apache.logging.log4j.Level
import org.scalatest.BeforeAndAfter
import org.scalatest.matchers.should.Matchers
import org.scalatest.time.{Seconds, Span}
@@ -30,8 +31,9 @@ import org.apache.spark.{SparkIllegalArgumentException,
TestUtils}
import org.apache.spark.sql._
import org.apache.spark.sql.connector.read.streaming
import org.apache.spark.sql.execution.streaming.checkpointing.{AsyncCommitLog,
AsyncOffsetSeqLog, CommitMetadata, OffsetSeq}
-import
org.apache.spark.sql.execution.streaming.runtime.{AsyncProgressTrackingMicroBatchExecution,
LongOffset, MemoryStream, StreamExecution}
+import
org.apache.spark.sql.execution.streaming.runtime.{AsyncProgressTrackingMicroBatchExecution,
LongOffset, MemoryStream, StreamExecution, StreamingQueryWrapper}
import
org.apache.spark.sql.execution.streaming.runtime.AsyncProgressTrackingMicroBatchExecution.{ASYNC_PROGRESS_TRACKING_CHECKPOINTING_INTERVAL_MS,
ASYNC_PROGRESS_TRACKING_ENABLED,
ASYNC_PROGRESS_TRACKING_OVERRIDE_SINK_SUPPORT_CHECK}
+import
org.apache.spark.sql.execution.streaming.state.{FailureInjectionCheckpointFileManager,
FailureInjectionFileSystem}
import org.apache.spark.sql.functions.{column, window}
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.streaming.{StreamingQuery,
StreamingQueryException, StreamTest, Trigger}
@@ -866,6 +868,61 @@ class AsyncProgressTrackingMicroBatchExecutionSuite
}
}
+ test("SPARK-60469: stop warns and still terminates when async log writes do
not finish") {
+ withSQLConf(
+ SQLConf.STREAMING_CHECKPOINT_FILE_MANAGER_CLASS.parent.key ->
+ classOf[FailureInjectionCheckpointFileManager].getName) {
+ withTempDir { checkpointDir =>
+ val injectionState =
FailureInjectionFileSystem.registerTempPath(checkpointDir.getPath)
+ val semaphore = FailureInjectionFileSystem.createAtomicDelaySemaphore
+ try {
+ val inputData = MemoryStream[Int]
+ val query = inputData.toDF().writeStream
+ .format("noop")
+ .option(ASYNC_PROGRESS_TRACKING_ENABLED, true)
+ .option(ASYNC_PROGRESS_TRACKING_CHECKPOINTING_INTERVAL_MS, 0)
+ .option("checkpointLocation", checkpointDir.getAbsolutePath)
+ .start()
+ inputData.addData(0)
+ query.processAllAvailable()
+
+ // Block the async commit log write of batch 1 so that it is still
running on stop.
+ injectionState.delayCreateAtomicRegex = Seq(".*/commits/1")
+ inputData.addData(1)
+ query.processAllAvailable()
+ eventually(timeout(streamingTimeout)) {
+ assert(semaphore.hasQueuedThreads)
+ }
+
+ val logAppender = new LogAppender("async progress tracking executor
shutdown")
+ logAppender.setThreshold(Level.WARN)
+ try {
+ withLogAppender(logAppender) {
+ // This waits 30 seconds for the blocked write.
+ query.stop()
+ }
+ } finally {
+ // Let the blocked write finish before the checkpoint directory is
deleted, and drop
+ // the permit if the write was interrupted instead.
+ injectionState.delayCreateAtomicRegex = Seq.empty
+ semaphore.release()
+
waitPendingOffsetWrites(query.asInstanceOf[StreamingQueryWrapper].streamingQuery)
+ semaphore.drainPermits()
+ }
+
assert(logAppender.loggingEvents.exists(_.getMessage.getFormattedMessage.endsWith(
+ "did not terminate in 30 seconds")))
+ assert(!spark.streams.active.exists(_.id == query.id))
+ } finally {
+ injectionState.delayCreateAtomicRegex = Seq.empty
+ if (semaphore.hasQueuedThreads) {
+ semaphore.release()
+ }
+
FailureInjectionFileSystem.removePathFromTempToInjectionState(checkpointDir.getPath)
+ }
+ }
+ }
+ }
+
test("commit intervals happy path") {
val checkpointLocation = Utils.createTempDir(namePrefix =
"streaming.metadata").getCanonicalPath
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]