Tim Meehan created SPARK-61259:
----------------------------------
Summary: Raise KeyboardInterrupt on SIGINT during a running action
in the Python client, as PySpark does
Key: SPARK-61259
URL: https://issues.apache.org/jira/browse/SPARK-61259
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Tim Meehan
In PySpark's Connect client, a SIGINT (Ctrl-C) during a long-running action
raises KeyboardInterrupt at once, and the session keeps working. In the Python
client of spark-connect-rust, the action keeps blocking and the signal is never
handled.
{code:python}
import os, signal, threading
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
slow = spark.range(0, 10**12, 1, 2).agg(F.sum(F.sqrt("id")))
threading.Timer(3.0, lambda: os.kill(os.getpid(), signal.SIGINT)).start()
slow.collect()
{code}
Against an OSS Spark 4.3.0 Connect server on loopback:
* pyspark-client 4.3.0: KeyboardInterrupt 3 seconds in; spark.range(3).count()
then returns 3.
* spark-connect-rust master 8768cc4: still blocked 12 seconds after the SIGINT.
At 4ad36945 it was still blocked after 60 seconds.
>From source: the action waits on the Tokio runtime with the GIL released, and
>nothing under crates/ calls Python::check_signals (PyErr_CheckSignals) while
>it waits, so Python's SIGINT handler never runs. SPARK-61250, which runs the
>synchronous API on another thread, did not change this. spark.interruptAll()
>from another thread does cancel the query.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
