Hyukjin Kwon created SPARK-62005:
------------------------------------
Summary: Let a signal end the wait of the exit hook of a Spark
Connect session, as PySpark's KeyboardInterrupt escapes
SparkConnectClient._on_exit
Key: SPARK-62005
URL: https://issues.apache.org/jira/browse/SPARK-62005
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
At interpreter exit, the exit hook of a session that is not stopped sends one
clean_cache command (SPARK-61753), and, with
SPARK_CONNECT_RELEASE_SESSION_ON_EXIT, releases and closes the session. When
the server is gone, the request waits on the retries of the retry policy, which
take over 15 minutes with the default policy, in both clients.
In pyspark-client 4.3.0 a signal ends that wait. SparkConnectClient._on_exit
(pyspark/sql/connect/client/core.py:2579-2597) calls _cleanup_ml_cache
(core.py:2598-2617), whose `except Exception: pass` does not catch the
KeyboardInterrupt of Ctrl-C: it is raised out of the sleep of the retries
(retries.py:306, `self._sleep(wait_time / 1000)`) and out of _on_exit, which
atexit reports ("Exception ignored in atexit callback") before it runs the next
hook. The session is then neither released nor closed.
The drop-in's exit hook (crates/pyspark-rs/src/session.rs,
PyExitHook::_on_exit) runs the requests in a bare `py.detach`, which never
checks the signals of the process, so Ctrl-C does nothing until the retries
end: the process can only be killed.
Repro (a session to a port where nothing listens, SIGINT 4 s into the exit;
probe p_exit_sigint.py):
from pyspark.sql.connect.session import SparkSession
spark = SparkSession.builder.remote("sc://localhost:<a port where nothing
listens>").getOrCreate()
# exit, then send SIGINT
- pyspark-client 4.3.0: exits 0.2 s after the SIGINT, status 0, with the
KeyboardInterrupt traceback in retries.py.
- drop-in: still waiting 45 s after the SIGINT (and over 900 s without one, as
pyspark-client).
Expected: the signal ends the wait as in PySpark: the exception of the signal
handler is raised out of the hook, the session is neither released nor closed,
and the process exits.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]