[
https://issues.apache.org/jira/browse/SPARK-58162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan resolved SPARK-58162.
---------------------------------
Fix Version/s: 4.3.0
Resolution: Fixed
Issue resolved by pull request 57320
[https://github.com/apache/spark/pull/57320]
> Spark Connect client-side unbounded RetryException retry loop in reattach
> path can hang forever
> -----------------------------------------------------------------------------------------------
>
> Key: SPARK-58162
> URL: https://issues.apache.org/jira/browse/SPARK-58162
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 4.1.0, 4.2.0, 4.3.0, 5.0.0
> Reporter: Vinod KC
> Assignee: Vinod KC
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0
>
> Attachments: spark-connect-retryexception-hang-demo.zip
>
>
> This is a Spark Connect client-side issue . The bug is entirely in the
> client's local retry/reattach handling logic (both the Scala and Python
> Connect clients).
> Spark Connect's client-side reattach logic retries an internal error marker
> (RetryException) with no limit — no retry count cap, no backoff, and no time
> bound.
> This marker is raised client-side when a reattach attempt keeps hitting
> `DEADLINE_EXCEEDED`, or when the initial query
> never reached the server at all.
> If the connection to the server becomes unable to make progress (e.g. a load
> balancer or NAT gateway silently drops it, or gRPC keepalive is disabled or
> hasn't detected the dead connection yet), the client retries this cycle
> forever, entirely on its own — the server is stateless in this exchange and
> does nothing wrong. The client never returns and never throws — a permanent
> hang, affecting both the Scala and Python clients identically.
> A minimal, runnable reproduction is attached:
> `spark-connect-retryexception-hang-demo.zip` a real Spark Connect server, a
> separate client process, and a small proxy that can be frozen to simulate a
> dropped connection. A thread dump confirms the client is permanently stuck
> retrying inside its own reattach retry loop, with no way out, while the
> server remains idle and unaffected throughout.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
