Hyukjin Kwon created SPARK-61953:
------------------------------------
Summary: Send no requests at exit for a session that the program
dropped, as PySpark's SparkSession closes its client when it is freed, and do
not keep its client until the exit
Key: SPARK-61953
URL: https://issues.apache.org/jira/browse/SPARK-61953
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
pyspark-client 4.3.0's SparkConnectClient registers
atexit.register(self._on_exit) (client/core.py:955-959), and _on_exit
(core.py:2579-2597) does nothing for a closed client. SparkSession.__del__
(connect/session.py:879-887) closes its client. So a session that the program
drops, for example s2 = spark.newSession(); ...; del s2, is freed by refcount,
and nothing is sent for it at exit.
The drop-in's exit hook (PyExitHook in crates/pyspark-rs/src/session.rs) held
the core session strongly. So the clean_cache of every session ever made was
sent at exit, even of the sessions the program dropped, and their clients and
channels lived until the exit. With the server gone, the exit then waited on
the retries: a script that dropped a newSession() hung at exit, where PySpark's
exits at once.
The exit hook should hold the session weakly. A session that no object of the
program can reach (its session objects, DataFrames, readers, writers, catalogs,
streams, ML objects) should send nothing at exit, and be dropped with its
client. A session still reachable at exit should send PySpark's _on_exit
requests, as now. A bare SparkConnectClient that no session object owns should
be kept until the exit, as PySpark's atexit keeps it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]