Hyukjin Kwon created SPARK-61353:
------------------------------------
Summary: Raise KeyboardInterrupt promptly on SIGINT during
SparkSession.stop and the native KMeans fit, in the Python client as PySpark
does
Key: SPARK-61353
URL: https://issues.apache.org/jira/browse/SPARK-61353
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
SPARK-61259 made the waits of the actions, writers, commands, streams, catalog
calls, ML fits and pipelines of the Python client raise the KeyboardInterrupt
of a SIGINT promptly, as PySpark does. Two waits still ignore it:
- SparkSession.stop(): it waits for the release of the session and for the
retries of that release (about 10 minutes with the default retry policy against
an unreachable server since SPARK-61319). PySpark's stop() raises
KeyboardInterrupt at once from the sleep between the retries, and leaves the
session open. A wait that nothing is executing on the server for cannot be
interrupted on the server, so the wait of SPARK-61259 keeps waiting until the
release ends.
- The fit of the native KMeans of SPARK-61229, which merged after SPARK-61259
was written and still waits without polling the signals.
Let a wait end at once when a signal handler raised and no execution is in
flight (Watch::Abort, honored by a new block_on_abortable that the release of
stop() uses), so that stop() raises the exception at once and leaves the
session open as PySpark's does, and make the KMeans fit interruptible as the
other fits are.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]