HyukjinKwon commented on code in PR #59277:
URL: https://github.com/apache/spark/pull/59277#discussion_r4253796565
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala:
##########
@@ -150,6 +150,29 @@ object Connect {
.intConf
.createWithDefaultString("1000")
+ val CONNECT_SESSION_ALLOW_RECONNECT_AFTER_CLOSE =
+ buildConf("spark.connect.session.allowReconnectAfterClose")
+ .doc(
+ "When true, the session id can be reused after this session is closed,
for example " +
+ "on inactivity expiry. The id stays reserved, and requests fail with
" +
+ "INVALID_HANDLE.SESSION_CLOSED, until the close finishes. After
that, a client that " +
+ "used the closed session gets INVALID_HANDLE.SESSION_CHANGED if it
sends its " +
+ "observed server-side session id, as the built-in Python and Scala
clients do. " +
+ "Such a client must create a new SparkSession with the same session
id. A client " +
+ "that does not send it silently gets a new session. The new session
is empty: " +
+ "server-side state " +
+ "such as registered UDFs, temporary views, temporary functions,
cached data and the " +
Review Comment:
**Nit (P3):** `SessionHolder.close()` doesn't uncache anything. Connect's
`persist()` goes through `target.persist()` in `SparkConnectAnalyzeHandler`,
which registers the plan in the `SharedState` `CacheManager` that every session
created with `newSession()` shares. So data cached with `df.cache()` or `CACHE
TABLE` in the closed session stays cached after the id is reused, and the new
session still hits it. For example, `spark.catalog.isCached` on a cached table
still returns `true`. Listing cached data here tells users that the close frees
it and that the new session starts uncached, and neither is true. Could we drop
it from this list and from the matching `docs/configuration.md` row?
```suggestion
"such as registered UDFs, temporary views, temporary functions and
the " +
```
**Verification:**
- **Inspection:** The conf doc string and the configuration reference row no
longer list cached data as state that is removed when the session closes, and
they describe the same lost-state list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
