[ 
https://issues.apache.org/jira/browse/SPARK-61894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61894:
-----------------------------------
    Labels: pull-request-available  (was: )

> Shuffle fetch hangs forever when a TransportResponseHandler success callback 
> throws
> -----------------------------------------------------------------------------------
>
>                 Key: SPARK-61894
>                 URL: https://issues.apache.org/jira/browse/SPARK-61894
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 5.0.0, 4.2.2
>            Reporter: Ivan Sadikov
>            Priority: Major
>              Labels: pull-request-available
>
> TransportResponseHandler.handle removes a ChunkFetchSuccess, RpcResponse or 
> MergedBlockMetaSuccess request from its outstanding requests before calling 
> the request's success callback. If the callback throws (for example an 
> OutOfMemoryError in the Netty event loop), the exception reaches 
> TransportChannelHandler.exceptionCaught, which fails the remaining 
> outstanding requests and closes the channel. The request whose callback threw 
> is no longer tracked, so neither the connection close nor the idle timeout 
> ever fails it, and its caller waits forever. On the ChunkFetchSuccess path 
> the response body is also not released.
> For a shuffle fetch, the block never reaches ShuffleBlockFetcherIterator's 
> results queue, so the task blocks on results.take() indefinitely: no 
> FetchFailedException, no retry, no task failure. The task only ends when it 
> is killed.
> With retries enabled (spark.shuffle.io.maxRetries > 0, the default), the same 
> hang happens one layer up: RetryingBlockTransferor removes the block from its 
> pending set before forwarding onBlockTransferSuccess. If the parent listener 
> throws, a failure reported for that block afterwards is ignored because the 
> block is no longer pending, so the parent never hears about it.
> This is the chunk/RPC follow-up for SPARK-57709, which fixed the same class 
> of hang for stream callbacks.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to