[
https://issues.apache.org/jira/browse/SPARK-58094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod KC updated SPARK-58094:
-----------------------------
Attachment: spark-connect-hang-demo.zip
> Blocking RPCs hang forever when the connection dies mid-wait (no gRPC
> keepalive / no client-side deadline)
> ----------------------------------------------------------------------------------------------------------
>
> Key: SPARK-58094
> URL: https://issues.apache.org/jira/browse/SPARK-58094
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 4.2.0, 4.3.0
> Reporter: Vinod KC
> Priority: Major
> Attachments: spark-connect-hang-demo.zip
>
>
> Any blocking call in Spark Connect can hang indefinitely and never throw if
> the network path between client and server goes silently dark while the RPC
> is waiting on a response (e.g. a NAT gateway, load balancer, or corporate
> proxy evicting an idle connection without sending a TCP RST/FIN). This is a
> transport-layer gap, not specific to any one API: Spark Connect configures no
> gRPC/HTTP2 keepalive on either side, and the client's blocking read path has
> no deadline by default. Any RPC whose response is delayed while its
> connection is silently dropped is exposed — most visibly
> `StreamingQuery.awaitTermination()` and other long-lived/idle-prone calls
> (long streaming queries, session status polling, gaps between reattach
> calls), since they hold a connection open and idle for long enough that a
> middlebox is likely to evict it — but the same failure mode applies to a
> regular (non-streaming) query's RPC if its connection happens to be silently
> dropped mid-wait; only the size of the exposure window differs, not the
> mechanism.
> *Impact:*
> A Mission Critical case surfaced via StreamingQuery.awaitTermination(): a
> streaming job stalled for hours after a single non-retryable source error.
> Symptoms:
> - StreamingQueryListener.onQueryTerminated() fired correctly with the
> terminal exception.
> - The client's blocked query.awaitTermination() call never returned and never
> threw.
> - The job run stayed "RUNNING" indefinitely; the driver process was healthy
> and idle the whole time (no CPU/network activity, no errors logged).
>
> *Root Cause:*
> Two independent gaps combine to produce the hang, both at the
> transport/channel level shared by all RPCs:
> 1. No gRPC/HTTP2 keepalive is configured on either the client channel
> (SparkConnectClient.scala) or the server (SparkConnectService.scala's
> NettyServerBuilder), so a silently-dropped connection is never detected.
> 2. The client's blocking read (ExecutePlanResponseReattachableIterator →
> ArrayBlockingQueue.take()) has no deadline, and the server's polling loops
> (e.g. handleStreamingAwaitTermination) only exit via
> Context.current.isCancelled, a signal that itself depends on the transport
> noticing the dead connection — which never happens without keepalive.
> *Issue repro:*
> Reproduced with two separate JVM processes (real Connect server + separate
> client), using a `frozen TCP proxy` to simulate a silently-dropped connection
> (no root/iptables needed).
> Confirmed via thread dump that the client is parked forever in
> ArrayBlockingQueue.take() while the server-side query has already terminated
> independently. awaitTermination() was used as the concrete repro vehicle
> because its long idle/blocking window makes the bug easy and reliable to
> trigger, not because the bug is scoped to it.
>
> Issue repro setup in zip file `spark-connect-hang-demo.zip`
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
