Hyukjin Kwon created SPARK-61953:
------------------------------------

             Summary: Send no requests at exit for a session that the program 
dropped, as PySpark's SparkSession closes its client when it is freed, and do 
not keep its client until the exit
                 Key: SPARK-61953
                 URL: https://issues.apache.org/jira/browse/SPARK-61953
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


pyspark-client 4.3.0's SparkConnectClient registers 
atexit.register(self._on_exit) (client/core.py:955-959), and _on_exit 
(core.py:2579-2597) does nothing for a closed client. SparkSession.__del__ 
(connect/session.py:879-887) closes its client. So a session that the program 
drops, for example s2 = spark.newSession(); ...; del s2, is freed by refcount, 
and nothing is sent for it at exit.

The drop-in's exit hook (PyExitHook in crates/pyspark-rs/src/session.rs) held 
the core session strongly. So the clean_cache of every session ever made was 
sent at exit, even of the sessions the program dropped, and their clients and 
channels lived until the exit. With the server gone, the exit then waited on 
the retries: a script that dropped a newSession() hung at exit, where PySpark's 
exits at once.

The exit hook should hold the session weakly. A session that no object of the 
program can reach (its session objects, DataFrames, readers, writers, catalogs, 
streams, ML objects) should send nothing at exit, and be dropped with its 
client. A session still reachable at exit should send PySpark's _on_exit 
requests, as now. A bare SparkConnectClient that no session object owns should 
be kept until the exit, as PySpark's atexit keeps it.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to