Venkata Sai Akhil Gudesa created SPARK-50710:
------------------------------------------------
Summary: Add support for optional client reconnection to sessions
after release
Key: SPARK-50710
URL: https://issues.apache.org/jira/browse/SPARK-50710
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: 4.0.0
Reporter: Venkata Sai Akhil Gudesa
Currently, the connect server will, by default, tombstone all sessions that
have either been released explicitly (through a `ReleaseSession` request) or
cleaned up due to inactivity/idleness in periodic checks.
Tombstoning prevents clients from reconnecting with the same `userId` and
`sessionId`. This mechanism ensures that clients do not accidentally end up
with a 'fresh' server-side session, which may be disastrous/fatal as all
previously held state is lost (e.g., Temporary views, temporary UDFs, modified
configs, current catalog, etc.).
Consider a client that runs simple non-state dependant queries (e.g `select
count(*) from ...`), perhaps sparsely during the lifetime of the application.
Such a client may prefer to opt out of tombstoning for the following reasons:
* Queries do not depend on any custom server-side state.
* Modifying `userId`/`sessionId` on each reconnect may be inconvenient for
tracking/observability purposes.
* On resource-constrained servers, clients may want to minimize their memory
footprint by explicitly releasing their state, especially when they believe
their requests are sparsely spread out.
Currently, the only way to allow clients to reconnect is to set `
spark.connect.session.manager.closedSessionsTombstonesSize` to `0`. However,
this is not ideal as it would allow *all* clients to reconnect, which as
previously pointed out, may be dangerous.
As an improvement, allowing specific clients to explicitly signal/request the
reconnection possibility addresses the needs mentioned earlier.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
