[
https://issues.apache.org/jira/browse/SPARK-61953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61953:
---------------------------------
Fix Version/s: connect-rust-4.4.0
> Send no requests at exit for a session that the program dropped, as PySpark's
> SparkSession closes its client when it is freed, and do not keep its client
> until the exit
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61953
> URL: https://issues.apache.org/jira/browse/SPARK-61953
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.4.0
>
>
> pyspark-client 4.3.0's SparkConnectClient registers
> atexit.register(self._on_exit) (client/core.py:955-959), and _on_exit
> (core.py:2579-2597) does nothing for a closed client. SparkSession.__del__
> (connect/session.py:879-887) closes its client. So a session that the program
> drops, for example s2 = spark.newSession(); ...; del s2, is freed by
> refcount, and nothing is sent for it at exit.
> The drop-in's exit hook (PyExitHook in crates/pyspark-rs/src/session.rs) held
> the core session strongly. So the clean_cache of every session ever made was
> sent at exit, even of the sessions the program dropped, and their clients and
> channels lived until the exit. With the server gone, the exit then waited on
> the retries: a script that dropped a newSession() hung at exit, where
> PySpark's exits at once.
> The exit hook should hold the session weakly. A session that no object of the
> program can reach (its session objects, DataFrames, readers, writers,
> catalogs, streams, ML objects) should send nothing at exit, and be dropped
> with its client. A session still reachable at exit should send PySpark's
> _on_exit requests, as now. A bare SparkConnectClient that no session object
> owns should be kept until the exit, as PySpark's atexit keeps it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]