Venkata krishnan Sowrirajan created SPARK-60062:
---------------------------------------------------
Summary: Allow recreating an expired Spark Connect session via
ReleaseSession(allow_reconnect=true)
Key: SPARK-60062
URL: https://issues.apache.org/jira/browse/SPARK-60062
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: 4.2.0
Reporter: Venkata krishnan Sowrirajan
When a Spark Connect session is closed, for example by inactivity expiry, its
session id is tombstoned in `closedSessionsCache`. Any later attempt to use
that id fails with `INVALID_HANDLE.SESSION_CLOSED`. The tombstone stays until
it's evicted from the size-bounded cache (
spark.connect.session.manager.closedSessionsTombstonesSize , default 1000),
which can take an arbitrarily long time on a quiet server.
Some clients derive a stable session id from a user/application identity so
that notebooks can reattach after a kernel restart. Once that session expires,
they can't knowingly start over under the same id. Calling
`ReleaseSession(allow_reconnect=true)` on the expired session does nothing,
because the session is no longer active. The only workaround is to switch to a
different session id.
The tombstone exists to stop accidental reuse of a closed session. A client
that explicitly accepts the state loss should be able to recreate it.
Proposal:
• If the session is already closed, `ReleaseSession` with
`allow_reconnect=true` removes the tombstone, so the same session id creates a
new, empty session.
• No proto field changes; the existing field's documentation is extended.
• Expose `allow_reconnect` in Python `SparkConnectClient.release_session` and
Scala `SparkConnectClient.releaseSession`.
• Default behaviour is unchanged.
Example:
{code:java}
spark.client.release_session(allow_reconnect=True)
spark =
SparkSession.builder.remote(f"sc://host:15002/;session_id={same_id}").create()
# new, empty session{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
