This is an automated email from the ASF dual-hosted git repository.
gengliangwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 415b03223a94 [SPARK-60246][CORE][FOLLOWUP] Remove logging from
FileDownloadCallback and test the close-during-write path
415b03223a94 is described below
commit 415b03223a94b91c2ba126edf795626279311bf0
Author: Chiran Ravani <[email protected]>
AuthorDate: Fri Oct 23 09:24:26 2026 -0700
[SPARK-60246][CORE][FOLLOWUP] Remove logging from FileDownloadCallback and
test the close-during-write path
### What changes were proposed in this pull request?
Follow-up to #59472.
- `FileDownloadCallback` no longer logs. Its callbacks run on the
files-client event loop, and logging there can block on a class loader that is
itself waiting on that event loop, for example `ExecutorClassLoader` fetching a
class over the same connection (SPARK-38446). A class-level comment on
`FileDownloadCallback` now states this rule for all of its callbacks. The
`discarding` flag only existed to log once, so it is removed too.
- The file download tests in `NettyRpcEnvSuite` set
`spark.files.io.numConnectionsPerPeer` to 1, so the downloads in a test always
share one connection. They are also prefixed with `SPARK-60246: `.
- New test `SPARK-60246: download callbacks drop data for a closed reader,
report other failures, and do not log`. It gets the real callbacks through a
mocked download client and checks three cases:
- When the reader closes while `onData` is blocked in the pipe write,
`onData` returns without throwing, and later data for that stream is dropped.
- When the sink fails while the reader is still open, `onData` throws
`ClosedChannelException`, so a real write failure is not hidden as a truncated
download.
- When the stream fails, `onFailure` passes its exact error to the reader.
It also checks that the callbacks log nothing.
### Why are the changes needed?
#59472 added debug logs on the files-client event loop, which can deadlock
as described above. Its early-close test closes the reader before data arrives,
so it does not reliably reach the `catch` for a reader that closes during a
write. No test covered the `!source.isOpen()` guard on that `catch`. The tests
also relied on the default of one connection per peer.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Ran the `SPARK-60246` tests in `NettyRpcEnvSuite`, `SslNettyRpcEnvSuite`,
and `AutoNettyRpcEnvSuite`. The new test fails if the `catch` guard in `onData`
is narrowed to `case _: IOException =>`, if `onFailure` stops passing its error
to the reader, or if a callback logs.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #59532 from cravani/followup-spark-60246.
Authored-by: Chiran Ravani <[email protected]>
Signed-off-by: Gengliang Wang <[email protected]>
---
.../org/apache/spark/rpc/netty/NettyRpcEnv.scala | 18 ++--
.../apache/spark/rpc/netty/NettyRpcEnvSuite.scala | 108 +++++++++++++++++++--
2 files changed, 108 insertions(+), 18 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala
b/core/src/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala
index 5602a7738765..177f42b694ec 100644
--- a/core/src/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala
+++ b/core/src/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala
@@ -442,14 +442,17 @@ private[netty] class NettyRpcEnv(
}
+ /**
+ * These callbacks run on the connection's event loop, so they must not log
(SPARK-38446).
+ * Log4j may load classes through this thread's context class loader, e.g.
to print a logged
+ * exception's stack trace. That loader can be an ExecutorClassLoader whose
lock is held by a
+ * task thread waiting for a class download that this event loop serves.
+ */
private class FileDownloadCallback(
sink: WritableByteChannel,
source: FileDownloadChannel,
client: TransportClient) extends StreamCallback {
- // Only accessed from the connection's event loop thread.
- private var discarding = false
-
override def onData(streamId: String, buf: ByteBuffer): Unit = {
// If the reader already closed its end (e.g. the fetch failed before
reading), drop the
// rest of this stream. Failing here would close the shared connection
and fail every
@@ -460,16 +463,9 @@ private[netty] class NettyRpcEnv(
sink.write(buf)
}
} catch {
- case e: IOException if !source.isOpen() =>
- logDebug(s"Reader of stream $streamId closed before reading it
fully; " +
- "discarding the rest of the stream", e)
- discarding = true
+ case _: IOException if !source.isOpen() =>
buf.position(buf.limit())
}
- } else if (!discarding) {
- logDebug(s"Reader of stream $streamId closed before reading it fully;
" +
- "discarding the rest of the stream")
- discarding = true
}
}
diff --git
a/core/src/test/scala/org/apache/spark/rpc/netty/NettyRpcEnvSuite.scala
b/core/src/test/scala/org/apache/spark/rpc/netty/NettyRpcEnvSuite.scala
index 2e546d250d65..242ca0871078 100644
--- a/core/src/test/scala/org/apache/spark/rpc/netty/NettyRpcEnvSuite.scala
+++ b/core/src/test/scala/org/apache/spark/rpc/netty/NettyRpcEnvSuite.scala
@@ -18,21 +18,27 @@
package org.apache.spark.rpc.netty
import java.io.{File, IOException}
-import java.nio.channels.Channels
+import java.nio.ByteBuffer
+import java.nio.channels.{Channels, ClosedChannelException,
ReadableByteChannel}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
import java.util.UUID
import java.util.concurrent.ExecutionException
+import scala.concurrent.{ExecutionContext, Future}
import scala.concurrent.duration._
-import org.mockito.Mockito.when
+import org.apache.logging.log4j.Level
+import org.mockito.ArgumentCaptor
+import org.mockito.ArgumentMatchers.{any, anyInt, eq => meq}
+import org.mockito.Mockito.{verify, when}
import org.scalatest.concurrent.{Signaler, ThreadSignaler, TimeLimits}
import org.scalatestplus.mockito.MockitoSugar
import org.apache.spark._
import org.apache.spark.deploy.SparkHadoopUtil
-import org.apache.spark.network.client.TransportClient
+import org.apache.spark.network.client.{StreamCallback, TransportClient,
TransportClientFactory}
+import org.apache.spark.network.util.JavaUtils
import org.apache.spark.rpc._
import org.apache.spark.util.{SslTestUtils, ThreadUtils, Utils}
@@ -218,7 +224,9 @@ class NettyRpcEnvSuite extends RpcEnvSuite with
MockitoSugar with TimeLimits {
private def withFileDownloadEnv(f: FileDownloadFixture => Unit): Unit = {
withTempDir { tempDir =>
withTempDir { destDir =>
- val conf = createSparkConf()
+ // Use a single file-download connection, so the downloads in a test
share one connection
+ // even if the JVM sets a higher connection count.
+ val conf =
createSparkConf().set("spark.files.io.numConnectionsPerPeer", "1")
val downloadEnv = createRpcEnv(conf, "file-download", 0)
val sparkEnv = SparkEnv.get
when(sparkEnv.rpcEnv).thenReturn(downloadEnv)
@@ -237,7 +245,7 @@ class NettyRpcEnvSuite extends RpcEnvSuite with
MockitoSugar with TimeLimits {
}
}
- test("a failed file fetch must not leave its download channel open") {
+ test("SPARK-60246: a failed file fetch must not leave its download channel
open") {
withFileDownloadEnv { d =>
// The target directory is missing, e.g. because it was deleted
concurrently, so the
// fetch fails creating its temp file after the download channel was
already opened.
@@ -249,7 +257,8 @@ class NettyRpcEnvSuite extends RpcEnvSuite with
MockitoSugar with TimeLimits {
}
}
- test("a download reader closing early must not fail other downloads on the
connection") {
+ test("SPARK-60246: a download reader closing early must not fail other
downloads on the " +
+ "connection") {
withFileDownloadEnv { d =>
// Open both downloads before the first reader goes away, so the second
one is already in
// flight on the same connection when the file-download thread finds the
first reader gone.
@@ -263,7 +272,8 @@ class NettyRpcEnvSuite extends RpcEnvSuite with
MockitoSugar with TimeLimits {
}
}
- test("a file fetch failing after the download keeps its error and the
connection usable") {
+ test("SPARK-60246: a file fetch failing after the download keeps its error
and the " +
+ "connection usable") {
withFileDownloadEnv { d =>
withTempDir { conflictDir =>
// A different file with the same name already exists and
spark.files.overwrite is off,
@@ -277,6 +287,90 @@ class NettyRpcEnvSuite extends RpcEnvSuite with
MockitoSugar with TimeLimits {
d.fetchSmallJar(d.destDir)
}
}
+
+ test("SPARK-60246: download callbacks drop data for a closed reader, report
other failures, " +
+ "and do not log") {
+ val downloadEnv =
+ createRpcEnv(createSparkConf(), "file-download-callback", 0, clientMode
= true)
+ val writerThreadName = "file-download-writer"
+ val writerPool =
ThreadUtils.newDaemonSingleThreadExecutor(writerThreadName)
+ try {
+ // Capture the real FileDownloadCallback without a server, so the test
controls exactly when
+ // data arrives. The client and its factory are mocked because the
callback can only be
+ // obtained from the `stream` call that openChannel makes on the
download client.
+ val client = mock[TransportClient]
+ val factory = mock[TransportClientFactory]
+ when(factory.createClient(any[String](), anyInt())).thenReturn(client)
+ val factoryField =
classOf[NettyRpcEnv].getDeclaredField("fileDownloadFactory")
+ factoryField.setAccessible(true)
+ factoryField.set(downloadEnv, factory)
+
+ def openDownload(path: String): (ReadableByteChannel, StreamCallback) = {
+ val source = downloadEnv.openChannel(s"spark://localhost:12345$path")
+ val callbackCaptor = ArgumentCaptor.forClass(classOf[StreamCallback])
+ verify(client).stream(meq(path), callbackCaptor.capture())
+ (source, callbackCaptor.getValue)
+ }
+ val (source, callback) = openDownload("/jars/big.jar")
+ val (otherSource, otherCallback) = openDownload("/jars/other.jar")
+ val (failedSource, failedCallback) = openDownload("/jars/failed.jar")
+ try {
+ // The callbacks run on the connection's event loop, so they must not
log (SPARK-38446).
+ // Here they run only on this thread and the writer thread, so only
events from those
+ // count; the suite's shared env logs under the same logger name.
+ val callbackThreads = Set(Thread.currentThread.getName,
writerThreadName)
+ val logAppender = new LogAppender("file download callbacks")
+ logAppender.setThreshold(Level.DEBUG)
+ withLogAppender(logAppender, loggerNames =
Seq(classOf[NettyRpcEnv].getName),
+ level = Some(Level.DEBUG)) {
+ // Much larger than an OS pipe buffer, so onData blocks in the pipe
write until the
+ // reader reads or closes its end.
+ val buf = ByteBuffer.allocate(4 * 1024 * 1024)
+ val writer = Future(callback.onData("big", buf))(
+ ExecutionContext.fromExecutorService(writerPool))
+
+ // Reading a byte proves onData is writing. It cannot have written
the whole buffer
+ // into the pipe, so it is still blocked writing when the reader
closes its end below.
+ failAfter(60.seconds) {
+ JavaUtils.readFully(source, ByteBuffer.allocate(1))
+ }
+ assert(buf.hasRemaining)
+ source.close()
+
+ // The callback must discard the rest of the buffer instead of
throwing, since
+ // throwing would close the connection shared with other downloads.
+ ThreadUtils.awaitResult(writer, 60.seconds)
+ // Later data for the same stream must not throw either.
+ callback.onData("big", ByteBuffer.allocate(1024))
+ callback.onComplete("big")
+
+ // A write that fails while the reader is still open is a real
failure and must
+ // propagate, so the reader sees an error instead of a silently
truncated download.
+ otherCallback.onComplete("other")
+ intercept[ClosedChannelException] {
+ otherCallback.onData("other", ByteBuffer.allocate(1024))
+ }
+
+ // A stream failure must reach the reader as the same exception.
+ val cause = new IOException("Stream failed")
+ failedCallback.onFailure("failed", cause)
+
assert(intercept[IOException](failedSource.read(ByteBuffer.allocate(1))) eq
cause)
+ }
+ val callbackEvents =
+ logAppender.loggingEvents.filter(e =>
callbackThreads.contains(e.getThreadName))
+ assert(callbackEvents.isEmpty)
+ } finally {
+ // Closing an already closed channel is a no-op, and onComplete closes
each sink.
+ Seq(source, otherSource, failedSource).foreach(_.close())
+ callback.onComplete("big")
+ otherCallback.onComplete("other")
+ failedCallback.onComplete("failed")
+ }
+ } finally {
+ ThreadUtils.shutdown(writerPool)
+ downloadEnv.shutdown()
+ }
+ }
}
class SslNettyRpcEnvSuite extends NettyRpcEnvSuite with MockitoSugar with
TimeLimits {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]