[
https://issues.apache.org/jira/browse/SPARK-61894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61894:
-----------------------------------
Labels: pull-request-available (was: )
> Shuffle fetch hangs forever when a TransportResponseHandler success callback
> throws
> -----------------------------------------------------------------------------------
>
> Key: SPARK-61894
> URL: https://issues.apache.org/jira/browse/SPARK-61894
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 5.0.0, 4.2.2
> Reporter: Ivan Sadikov
> Priority: Major
> Labels: pull-request-available
>
> TransportResponseHandler.handle removes a ChunkFetchSuccess, RpcResponse or
> MergedBlockMetaSuccess request from its outstanding requests before calling
> the request's success callback. If the callback throws (for example an
> OutOfMemoryError in the Netty event loop), the exception reaches
> TransportChannelHandler.exceptionCaught, which fails the remaining
> outstanding requests and closes the channel. The request whose callback threw
> is no longer tracked, so neither the connection close nor the idle timeout
> ever fails it, and its caller waits forever. On the ChunkFetchSuccess path
> the response body is also not released.
> For a shuffle fetch, the block never reaches ShuffleBlockFetcherIterator's
> results queue, so the task blocks on results.take() indefinitely: no
> FetchFailedException, no retry, no task failure. The task only ends when it
> is killed.
> With retries enabled (spark.shuffle.io.maxRetries > 0, the default), the same
> hang happens one layer up: RetryingBlockTransferor removes the block from its
> pending set before forwarding onBlockTransferSuccess. If the parent listener
> throws, a failure reported for that block afterwards is ignored because the
> block is no longer pending, so the parent never hears about it.
> This is the chunk/RPC follow-up for SPARK-57709, which fixed the same class
> of hang for stream callbacks.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]