Tim Meehan created SPARK-61410:
----------------------------------

             Summary: Raise NO_ACTIVE_SESSION from the first call after the 
server-side session changed, as PySpark does
                 Key: SPARK-61410
                 URL: https://issues.apache.org/jira/browse/SPARK-61410
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Tim Meehan


When the server-side session changes, the first call of the Rust client raises 
the server's INVALID_HANDLE.SESSION_CHANGED (an UnknownException in the Python 
client of spark-connect-rust), and the calls after it raise NO_ACTIVE_SESSION. 
PySpark raises NO_ACTIVE_SESSION from the first call. Only the first call's 
error differs.

The change is provoked without a restart: the client's observed server-side 
session ID is set to another UUID, so the request carries an ID the server does 
not hold, and the server answers with SESSION_CHANGED, as after a restart.

{code:python}
import uuid
from pyspark.sql.connect.session import SparkSession

spark = SparkSession(connection="sc://localhost:15002")
spark.range(3).count()
spark._client._server_session_id = str(uuid.uuid4())
for _ in range(2):
    try:
        spark.range(3).collect()
    except Exception as e:
        print(type(e).__name__, e.getCondition())
{code}

Against an OSS Spark 4.3.0 Connect server on loopback:
* pyspark-client 4.3.0: first call SparkConnectException NO_ACTIVE_SESSION; 
second call SparkConnectException NO_ACTIVE_SESSION.
* spark-connect-rust master 125654b: first call UnknownException 
INVALID_HANDLE.SESSION_CHANGED; second call SparkConnectException 
NO_ACTIVE_SESSION.

>From source, at 125654b: SparkConnectClient::grpc_error closes the client when 
>it sees INVALID_HANDLE.SESSION_CHANGED 
>(crates/spark-connect-core/src/client.rs:705-709). error_from_status then 
>fetches the details of the error under its errorId through error_details, 
>which does not check that the client is open, and falls back to the bare error 
>if the fetch fails (client.rs:1268-1279, 1286-1305). PySpark does the same in 
>_handle_rpc_error, but its _fetch_enriched_error goes through _stub, which 
>raises NO_ACTIVE_SESSION once the client is closed 
>(pyspark/sql/connect/client/core.py:961-969, 2349-2368, 2429-2447). That 
>exception is not a grpc.RpcError, so it is not caught and it replaces the 
>error of the server.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to