Hyukjin Kwon created SPARK-59032:
------------------------------------
Summary: Connect parity CI hangs on test_parity_arrow_python_udf.py
Key: SPARK-59032
URL: https://issues.apache.org/jira/browse/SPARK-59032
Project: Spark
Issue Type: Bug
Components: Project Infra
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
The Python Connect parity gate hangs on test_parity_arrow_python_udf.py on the
CI runner: the whole file times out with p=0 (zero tests completed), even
though the exact same file passes deterministically locally in ~113s (320
passed, 64 skipped), matching the reference client's ~120s.
Root cause: on the loaded single-node CI server, a slow first arrow-Python-UDF
worker lets the default reattachable-execute senderMaxStreamDuration elapse, so
the server ends the ExecutePlan response stream early and the client must
ReattachExecute; that reattach path stalls under these conditions and hangs the
whole file.
Fix: start the CI Connect server with
spark.connect.execute.reattachable.senderMaxStreamDuration=1200s so the stream
stays open until results are produced and no reattach is triggered. (Hardening
the client reattach path is worth a separate follow-up.)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
