Ivan Sadikov created SPARK-61894:
------------------------------------

             Summary: Shuffle fetch hangs forever when a 
TransportResponseHandler success callback throws
                 Key: SPARK-61894
                 URL: https://issues.apache.org/jira/browse/SPARK-61894
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 4.2.2, 5.0.0
            Reporter: Ivan Sadikov


TransportResponseHandler.handle removes a ChunkFetchSuccess, RpcResponse or 
MergedBlockMetaSuccess request from its outstanding requests before calling the 
request's success callback. If the callback throws (for example an 
OutOfMemoryError in the Netty event loop), the exception reaches 
TransportChannelHandler.exceptionCaught, which fails the remaining outstanding 
requests and closes the channel. The request whose callback threw is no longer 
tracked, so neither the connection close nor the idle timeout ever fails it, 
and its caller waits forever. On the ChunkFetchSuccess path the response body 
is also not released.

For a shuffle fetch, the block never reaches ShuffleBlockFetcherIterator's 
results queue, so the task blocks on results.take() indefinitely: no 
FetchFailedException, no retry, no task failure. The task only ends when it is 
killed.

With retries enabled (spark.shuffle.io.maxRetries > 0, the default), the same 
hang happens one layer up: RetryingBlockTransferor removes the block from its 
pending set before forwarding onBlockTransferSuccess. If the parent listener 
throws, a failure reported for that block afterwards is ignored because the 
block is no longer pending, so the parent never hears about it.

This is the chunk/RPC follow-up for SPARK-57709, which fixed the same class of 
hang for stream callbacks.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to