[
https://issues.apache.org/jira/browse/SPARK-59032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-59032:
---------------------------------
Description:
The Python Connect parity gate intermittently hangs on
test_parity_arrow_python_udf.py: the file times out with p=0 (not one test
completed) and is killed only by the per-file cap. It was observed once on
master at file 5/84 on a fresh server, while all 83 other files (including the
sibling arrow/UDF files) passed.
It is NOT deterministic and NOT a slowness/timeout problem:
- the file passes standalone in ~122s (matching the reference client);
- it passes even with the suite's reattach-forcing server limits
(senderMaxStreamDuration=1s, senderMaxStreamSize=123);
- it did NOT reproduce in 8 back-to-back runs under the exact gate invocation;
- a temporary CI diagnostic (client py-spy + faulthandler, server jstack,
Python-worker stacks) found no deterministic hang.
Root cause: the client never applies gRPC keepalive to its channel. The
connection string's grpc_keepalive_* options (default: enabled, 60s interval /
20s timeout, keep-alive-while-idle) are parsed by ChannelBuilder but were never
applied to the tonic channel, and no HTTP/2 keepalive was set otherwise. So a
server-streaming RPC whose connection wedges (a half-open TCP, or a server-side
stall) blocks in stream.message() indefinitely: nothing pings the peer to
surface the break, the reattachable-execute iterator never gets the transport
error it needs to reattach, and the stream only unblocks when the client is
closed (at CI teardown).
Fix (PR: https://github.com/apache/spark-connect-rust/pull/58):
- Apply the parsed keepalive settings to the tonic endpoint (TLS and plaintext
paths), so an unacked PING within the timeout fails the stream with a transport
error that the reattachable iterator treats as retriable and recovers from.
- Retry the file in the parity gate (FLAKY_FILES) as a belt-and-suspenders
safety net for any residual rarity, instead of skipping the whole file or
letting a rare hang burn the cap and fail.
- Reverted the earlier, wrong senderMaxStreamDuration=1200s server-conf attempt
(a larger cap only makes a hang wait longer), and removed the temporary
diagnostic workflow.
was:
The Python Connect parity gate hangs on test_parity_arrow_python_udf.py on the
CI runner: the whole file times out with p=0 (zero tests completed), even
though the exact same file passes deterministically locally in ~113s (320
passed, 64 skipped), matching the reference client's ~120s.
Root cause: on the loaded single-node CI server, a slow first arrow-Python-UDF
worker lets the default reattachable-execute senderMaxStreamDuration elapse, so
the server ends the ExecutePlan response stream early and the client must
ReattachExecute; that reattach path stalls under these conditions and hangs the
whole file.
Fix: start the CI Connect server with
spark.connect.execute.reattachable.senderMaxStreamDuration=1200s so the stream
stays open until results are produced and no reattach is triggered. (Hardening
the client reattach path is worth a separate follow-up.)
Summary: Spark Connect client can hang on a wedged stream: gRPC
keepalive is parsed but never applied (was: Connect parity CI hangs on
test_parity_arrow_python_udf.py)
> Spark Connect client can hang on a wedged stream: gRPC keepalive is parsed
> but never applied
> --------------------------------------------------------------------------------------------
>
> Key: SPARK-59032
> URL: https://issues.apache.org/jira/browse/SPARK-59032
> Project: Spark
> Issue Type: Bug
> Components: Project Infra
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Priority: Major
>
> The Python Connect parity gate intermittently hangs on
> test_parity_arrow_python_udf.py: the file times out with p=0 (not one test
> completed) and is killed only by the per-file cap. It was observed once on
> master at file 5/84 on a fresh server, while all 83 other files (including
> the sibling arrow/UDF files) passed.
> It is NOT deterministic and NOT a slowness/timeout problem:
> - the file passes standalone in ~122s (matching the reference client);
> - it passes even with the suite's reattach-forcing server limits
> (senderMaxStreamDuration=1s, senderMaxStreamSize=123);
> - it did NOT reproduce in 8 back-to-back runs under the exact gate invocation;
> - a temporary CI diagnostic (client py-spy + faulthandler, server jstack,
> Python-worker stacks) found no deterministic hang.
> Root cause: the client never applies gRPC keepalive to its channel. The
> connection string's grpc_keepalive_* options (default: enabled, 60s interval
> / 20s timeout, keep-alive-while-idle) are parsed by ChannelBuilder but were
> never applied to the tonic channel, and no HTTP/2 keepalive was set
> otherwise. So a server-streaming RPC whose connection wedges (a half-open
> TCP, or a server-side stall) blocks in stream.message() indefinitely: nothing
> pings the peer to surface the break, the reattachable-execute iterator never
> gets the transport error it needs to reattach, and the stream only unblocks
> when the client is closed (at CI teardown).
> Fix (PR: https://github.com/apache/spark-connect-rust/pull/58):
> - Apply the parsed keepalive settings to the tonic endpoint (TLS and
> plaintext paths), so an unacked PING within the timeout fails the stream with
> a transport error that the reattachable iterator treats as retriable and
> recovers from.
> - Retry the file in the parity gate (FLAKY_FILES) as a belt-and-suspenders
> safety net for any residual rarity, instead of skipping the whole file or
> letting a rare hang burn the cap and fail.
> - Reverted the earlier, wrong senderMaxStreamDuration=1200s server-conf
> attempt (a larger cap only makes a hang wait longer), and removed the
> temporary diagnostic workflow.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
