[
https://issues.apache.org/jira/browse/SPARK-59125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaicheng Zhou updated SPARK-59125:
----------------------------------
Description:
SparkConnectClient._on_exit() unconditionally invokes _cleanup_ml_cache() for
clients that remain open. If release_session() was called explicitly, the
server-side session has never been created, or has already been released, but
the client is not marked as closed. Consequently, the exit handler sends an
unnecessary ML cache cleanup RPC to the unalive session. The client should
record a successful session release and make _on_exit() a no-op afterward. This
avoids redundant RPCs and potential delays during interpreter shutdown.
Following up on SPARK-55326.
was:
SparkConnectClient._on_exit() unconditionally invokes _cleanup_ml_cache() for
clients that remain open. If release_session() was called explicitly, the
server-side session has already been released, but the client is not marked as
closed. Consequently, the exit handler sends an unnecessary ML cache cleanup
RPC to the released session. The client should record a successful session
release and make _on_exit() a no-op afterward. This avoids redundant RPCs and
potential delays during interpreter shutdown.
Following up on SPARK-55326.
Summary: Spark Connect sends cleanup RPC after the session is released
or is never created (was: Spark Connect sends cleanup RPC after the session is
released)
> Spark Connect sends cleanup RPC after the session is released or is never
> created
> ---------------------------------------------------------------------------------
>
> Key: SPARK-59125
> URL: https://issues.apache.org/jira/browse/SPARK-59125
> Project: Spark
> Issue Type: Bug
> Components: Connect, PySpark
> Affects Versions: 4.2.0, 4.3.0, 4.4.0
> Reporter: Kaicheng Zhou
> Priority: Major
> Labels: pull-request-available
>
> SparkConnectClient._on_exit() unconditionally invokes _cleanup_ml_cache() for
> clients that remain open. If release_session() was called explicitly, the
> server-side session has never been created, or has already been released, but
> the client is not marked as closed. Consequently, the exit handler sends an
> unnecessary ML cache cleanup RPC to the unalive session. The client should
> record a successful session release and make _on_exit() a no-op afterward.
> This avoids redundant RPCs and potential delays during interpreter shutdown.
> Following up on SPARK-55326.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
