Hyukjin Kwon created SPARK-61353:
------------------------------------

             Summary: Raise KeyboardInterrupt promptly on SIGINT during 
SparkSession.stop and the native KMeans fit, in the Python client as PySpark 
does
                 Key: SPARK-61353
                 URL: https://issues.apache.org/jira/browse/SPARK-61353
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


SPARK-61259 made the waits of the actions, writers, commands, streams, catalog 
calls, ML fits and pipelines of the Python client raise the KeyboardInterrupt 
of a SIGINT promptly, as PySpark does. Two waits still ignore it:
- SparkSession.stop(): it waits for the release of the session and for the 
retries of that release (about 10 minutes with the default retry policy against 
an unreachable server since SPARK-61319). PySpark's stop() raises 
KeyboardInterrupt at once from the sleep between the retries, and leaves the 
session open. A wait that nothing is executing on the server for cannot be 
interrupted on the server, so the wait of SPARK-61259 keeps waiting until the 
release ends.
- The fit of the native KMeans of SPARK-61229, which merged after SPARK-61259 
was written and still waits without polling the signals.

Let a wait end at once when a signal handler raised and no execution is in 
flight (Watch::Abort, honored by a new block_on_abortable that the release of 
stop() uses), so that stop() raises the exception at once and leaves the 
session open as PySpark's does, and make the KMeans fit interruptible as the 
other fits are.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to