Ivan Sadikov created SPARK-61894:
------------------------------------
Summary: Shuffle fetch hangs forever when a
TransportResponseHandler success callback throws
Key: SPARK-61894
URL: https://issues.apache.org/jira/browse/SPARK-61894
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 4.2.2, 5.0.0
Reporter: Ivan Sadikov
TransportResponseHandler.handle removes a ChunkFetchSuccess, RpcResponse or
MergedBlockMetaSuccess request from its outstanding requests before calling the
request's success callback. If the callback throws (for example an
OutOfMemoryError in the Netty event loop), the exception reaches
TransportChannelHandler.exceptionCaught, which fails the remaining outstanding
requests and closes the channel. The request whose callback threw is no longer
tracked, so neither the connection close nor the idle timeout ever fails it,
and its caller waits forever. On the ChunkFetchSuccess path the response body
is also not released.
For a shuffle fetch, the block never reaches ShuffleBlockFetcherIterator's
results queue, so the task blocks on results.take() indefinitely: no
FetchFailedException, no retry, no task failure. The task only ends when it is
killed.
With retries enabled (spark.shuffle.io.maxRetries > 0, the default), the same
hang happens one layer up: RetryingBlockTransferor removes the block from its
pending set before forwarding onBlockTransferSuccess. If the parent listener
throws, a failure reported for that block afterwards is ignored because the
block is no longer pending, so the parent never hears about it.
This is the chunk/RPC follow-up for SPARK-57709, which fixed the same class of
hang for stream callbacks.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]