[
https://issues.apache.org/jira/browse/SPARK-59125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaicheng Zhou updated SPARK-59125:
----------------------------------
Description:
SparkConnectClient._on_exit() unconditionally invokes _cleanup_ml_cache() for
clients that remain open. If release_session() was called explicitly, the
server-side session has already been released, but the client is not marked as
closed. Consequently, the exit handler sends an unnecessary ML cache cleanup
RPC to the released session. The client should record a successful session
release and make _on_exit() a no-op afterward. This avoids redundant RPCs and
potential delays during interpreter shutdown.
Following up on SPARK-55326.
was:SparkConnectClient._on_exit() unconditionally invokes _cleanup_ml_cache()
for clients that remain open. If release_session() was called explicitly, the
server-side session has already been released, but the client is not marked as
closed. Consequently, the exit handler sends an unnecessary ML cache cleanup
RPC to the released session. The client should record a successful session
release and make _on_exit() a no-op afterward. This avoids redundant RPCs and
potential delays during interpreter shutdown.
> Spark Connect sends cleanup RPC after the session is released
> -------------------------------------------------------------
>
> Key: SPARK-59125
> URL: https://issues.apache.org/jira/browse/SPARK-59125
> Project: Spark
> Issue Type: Bug
> Components: Connect, PySpark
> Affects Versions: 4.2.0, 4.3.0, 4.4.0
> Reporter: Kaicheng Zhou
> Priority: Major
>
> SparkConnectClient._on_exit() unconditionally invokes _cleanup_ml_cache() for
> clients that remain open. If release_session() was called explicitly, the
> server-side session has already been released, but the client is not marked
> as closed. Consequently, the exit handler sends an unnecessary ML cache
> cleanup RPC to the released session. The client should record a successful
> session release and make _on_exit() a no-op afterward. This avoids redundant
> RPCs and potential delays during interpreter shutdown.
> Following up on SPARK-55326.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
