Hyukjin Kwon created SPARK-62005:
------------------------------------

             Summary: Let a signal end the wait of the exit hook of a Spark 
Connect session, as PySpark's KeyboardInterrupt escapes 
SparkConnectClient._on_exit
                 Key: SPARK-62005
                 URL: https://issues.apache.org/jira/browse/SPARK-62005
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


At interpreter exit, the exit hook of a session that is not stopped sends one 
clean_cache command (SPARK-61753), and, with 
SPARK_CONNECT_RELEASE_SESSION_ON_EXIT, releases and closes the session. When 
the server is gone, the request waits on the retries of the retry policy, which 
take over 15 minutes with the default policy, in both clients.

In pyspark-client 4.3.0 a signal ends that wait. SparkConnectClient._on_exit 
(pyspark/sql/connect/client/core.py:2579-2597) calls _cleanup_ml_cache 
(core.py:2598-2617), whose `except Exception: pass` does not catch the 
KeyboardInterrupt of Ctrl-C: it is raised out of the sleep of the retries 
(retries.py:306, `self._sleep(wait_time / 1000)`) and out of _on_exit, which 
atexit reports ("Exception ignored in atexit callback") before it runs the next 
hook. The session is then neither released nor closed.

The drop-in's exit hook (crates/pyspark-rs/src/session.rs, 
PyExitHook::_on_exit) runs the requests in a bare `py.detach`, which never 
checks the signals of the process, so Ctrl-C does nothing until the retries 
end: the process can only be killed.

Repro (a session to a port where nothing listens, SIGINT 4 s into the exit; 
probe p_exit_sigint.py):

    from pyspark.sql.connect.session import SparkSession
    spark = SparkSession.builder.remote("sc://localhost:<a port where nothing 
listens>").getOrCreate()
    # exit, then send SIGINT

- pyspark-client 4.3.0: exits 0.2 s after the SIGINT, status 0, with the 
KeyboardInterrupt traceback in retries.py.
- drop-in: still waiting 45 s after the SIGINT (and over 900 s without one, as 
pyspark-client).

Expected: the signal ends the wait as in PySpark: the exception of the signal 
handler is raised out of the hook, the session is neither released nor closed, 
and the process exits.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to