Chiran Ravani created SPARK-60246:
-------------------------------------
Summary: Failed spark:// fetch blocks all later file downloads on
the executor
Key: SPARK-60246
URL: https://issues.apache.org/jira/browse/SPARK-60246
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 4.1.3, 4.3.0
Reporter: Chiran Ravani
When `Utils.fetchFile` downloads a `spark://` URL, `doFetchFile` opens the RPC
download channel and then calls `downloadFile`. `downloadFile` creates its temp
file before its try block. If that fails, for example because the target
directory was deleted, the method throws and the input stream is never closed.
The driver keeps sending the file. On the receiving side,
`FileDownloadCallback.onData` writes the bytes into a pipe from the
file-download Netty thread. `spark.files.io.threads` defaults to 1, so there is
one such thread. Nobody reads the pipe, so once it is full the write blocks
forever.
Every later download through that `RpcEnv` waits behind the blocked thread:
added jars and files, and REPL classes loaded by `ExecutorClassLoader`. Tasks
that need any of them hang. There is no timeout on this path.
The stuck thread:
{code}
"files-client-..." RUNNABLE
sun.nio.ch.FileDispatcherImpl.write0(Native Method)
sun.nio.ch.SinkChannelImpl.write(SinkChannelImpl.java:273)
org.apache.spark.rpc.netty.NettyRpcEnv$FileDownloadCallback.onData(NettyRpcEnv.scala)
org.apache.spark.network.client.StreamInterceptor.handle(StreamInterceptor.java)
org.apache.spark.network.util.TransportFrameDecoder.feedInterceptor(TransportFrameDecoder.java)
{code}
Closing the stream on failure is not enough by itself. The Netty thread's next
write then fails with "IOException: Broken pipe". That closes the connection
and fails every other download in flight on it.
To reproduce: fetch a large file (16 MiB) from a `spark://` URL into a
directory that does not exist, then fetch a small file through the same
`RpcEnv`. The second fetch never completes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
