Tim Meehan created SPARK-61410:
----------------------------------
Summary: Raise NO_ACTIVE_SESSION from the first call after the
server-side session changed, as PySpark does
Key: SPARK-61410
URL: https://issues.apache.org/jira/browse/SPARK-61410
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Tim Meehan
When the server-side session changes, the first call of the Rust client raises
the server's INVALID_HANDLE.SESSION_CHANGED (an UnknownException in the Python
client of spark-connect-rust), and the calls after it raise NO_ACTIVE_SESSION.
PySpark raises NO_ACTIVE_SESSION from the first call. Only the first call's
error differs.
The change is provoked without a restart: the client's observed server-side
session ID is set to another UUID, so the request carries an ID the server does
not hold, and the server answers with SESSION_CHANGED, as after a restart.
{code:python}
import uuid
from pyspark.sql.connect.session import SparkSession
spark = SparkSession(connection="sc://localhost:15002")
spark.range(3).count()
spark._client._server_session_id = str(uuid.uuid4())
for _ in range(2):
try:
spark.range(3).collect()
except Exception as e:
print(type(e).__name__, e.getCondition())
{code}
Against an OSS Spark 4.3.0 Connect server on loopback:
* pyspark-client 4.3.0: first call SparkConnectException NO_ACTIVE_SESSION;
second call SparkConnectException NO_ACTIVE_SESSION.
* spark-connect-rust master 125654b: first call UnknownException
INVALID_HANDLE.SESSION_CHANGED; second call SparkConnectException
NO_ACTIVE_SESSION.
>From source, at 125654b: SparkConnectClient::grpc_error closes the client when
>it sees INVALID_HANDLE.SESSION_CHANGED
>(crates/spark-connect-core/src/client.rs:705-709). error_from_status then
>fetches the details of the error under its errorId through error_details,
>which does not check that the client is open, and falls back to the bare error
>if the fetch fails (client.rs:1268-1279, 1286-1305). PySpark does the same in
>_handle_rpc_error, but its _fetch_enriched_error goes through _stub, which
>raises NO_ACTIVE_SESSION once the client is closed
>(pyspark/sql/connect/client/core.py:961-969, 2349-2368, 2429-2447). That
>exception is not a grpc.RpcError, so it is not caught and it replaces the
>error of the server.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]