Hyukjin Kwon created SPARK-61198:
------------------------------------
Summary: Raise NO_ACTIVE_SESSION in the retries of an RPC whose
session was stopped, in the Rust client as PySpark does
Key: SPARK-61198
URL: https://issues.apache.org/jira/browse/SPARK-61198
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark's retry loop sends each attempt of an RPC through
SparkConnectClient._stub, which raises NO_ACTIVE_SESSION once the client is
closed. So a call that waits to retry (for example an ExecutePlan that got
UNAVAILABLE) when its session is stopped raises NO_ACTIVE_SESSION at its next
attempt and sends nothing more.
The Rust client checks that the client is open only when a call starts, and in
the reattach of a result stream (SPARK-61187). Its retry loop (with_retry)
sends every attempt after each backoff without checking, so a call of a stopped
session keeps calling the server until the retry policy gives up, which takes
minutes with the default policy. The fix checks the client at each retry, as
the reference's stub does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
