[
https://issues.apache.org/jira/browse/SPARK-60246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gengliang Wang reassigned SPARK-60246:
--------------------------------------
Assignee: Chiran Ravani
> Failed spark:// fetch blocks all later file downloads on the executor
> ---------------------------------------------------------------------
>
> Key: SPARK-60246
> URL: https://issues.apache.org/jira/browse/SPARK-60246
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 4.3.0, 4.1.3
> Reporter: Chiran Ravani
> Assignee: Chiran Ravani
> Priority: Major
> Labels: pull-request-available
>
> When `Utils.fetchFile` downloads a `spark://` URL, `doFetchFile` opens the
> RPC download channel and then calls `downloadFile`. `downloadFile` creates
> its temp file before its try block. If that fails, for example because the
> target directory was deleted, the method throws and the input stream is never
> closed.
> The driver keeps sending the file. On the receiving side,
> `FileDownloadCallback.onData` writes the bytes into a pipe from the
> file-download Netty thread. `spark.files.io.threads` defaults to 1, so there
> is one such thread. Nobody reads the pipe, so once it is full the write
> blocks forever.
> Every later download through that `RpcEnv` waits behind the blocked thread:
> added jars and files, and REPL classes loaded by `ExecutorClassLoader`. Tasks
> that need any of them hang. There is no timeout on this path.
> The stuck thread:
> {code}
> "files-client-..." RUNNABLE
> sun.nio.ch.FileDispatcherImpl.write0(Native Method)
> sun.nio.ch.SinkChannelImpl.write(SinkChannelImpl.java:273)
>
> org.apache.spark.rpc.netty.NettyRpcEnv$FileDownloadCallback.onData(NettyRpcEnv.scala)
>
> org.apache.spark.network.client.StreamInterceptor.handle(StreamInterceptor.java)
>
> org.apache.spark.network.util.TransportFrameDecoder.feedInterceptor(TransportFrameDecoder.java)
> {code}
> Closing the stream on failure is not enough by itself. The Netty thread's
> next write then fails with "IOException: Broken pipe". That closes the
> connection and fails every other download in flight on it.
> To reproduce: fetch a large file (16 MiB) from a `spark://` URL into a
> directory that does not exist, then fetch a small file through the same
> `RpcEnv`. The second fetch never completes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
