[ 
https://issues.apache.org/jira/browse/SPARK-62005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-62005:
-----------------------------------
    Labels: pull-request-available  (was: )

> Let a signal end the wait of the exit hook of a Spark Connect session, as 
> PySpark's KeyboardInterrupt escapes SparkConnectClient._on_exit
> -----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62005
>                 URL: https://issues.apache.org/jira/browse/SPARK-62005
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>
> At interpreter exit, the exit hook of a session that is not stopped sends one 
> clean_cache command (SPARK-61753), and, with 
> SPARK_CONNECT_RELEASE_SESSION_ON_EXIT, releases and closes the session. When 
> the server is gone, the request waits on the retries of the retry policy, 
> which take over 15 minutes with the default policy, in both clients.
> In pyspark-client 4.3.0 a signal ends that wait. SparkConnectClient._on_exit 
> (pyspark/sql/connect/client/core.py:2579-2597) calls _cleanup_ml_cache 
> (core.py:2598-2617), whose `except Exception: pass` does not catch the 
> KeyboardInterrupt of Ctrl-C: it is raised out of the sleep of the retries 
> (retries.py:306, `self._sleep(wait_time / 1000)`) and out of _on_exit, which 
> atexit reports ("Exception ignored in atexit callback") before it runs the 
> next hook. The session is then neither released nor closed.
> The drop-in's exit hook (crates/pyspark-rs/src/session.rs, 
> PyExitHook::_on_exit) runs the requests in a bare `py.detach`, which never 
> checks the signals of the process, so Ctrl-C does nothing until the retries 
> end: the process can only be killed.
> Repro (a session to a port where nothing listens, SIGINT 4 s into the exit; 
> probe p_exit_sigint.py):
>     from pyspark.sql.connect.session import SparkSession
>     spark = SparkSession.builder.remote("sc://localhost:<a port where nothing 
> listens>").getOrCreate()
>     # exit, then send SIGINT
> - pyspark-client 4.3.0: exits 0.2 s after the SIGINT, status 0, with the 
> KeyboardInterrupt traceback in retries.py.
> - drop-in: still waiting 45 s after the SIGINT (and over 900 s without one, 
> as pyspark-client).
> Expected: the signal ends the wait as in PySpark: the exception of the signal 
> handler is raised out of the hook, the session is neither released nor 
> closed, and the process exits.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to