This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new bb1f5a19 [SPARK-61188][CONNECT][PYTHON] Raise RecursionError instead 
of crashing when progress handlers run queries without end, in the Python 
client as PySpark does
bb1f5a19 is described below

commit bb1f5a1931e7e37335046a071edb55c048c3d917
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 19:41:08 2026 +0900

    [SPARK-61188][CONNECT][PYTHON] Raise RecursionError instead of crashing 
when progress handlers run queries without end, in the Python client as PySpark 
does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61188](https://issues.apache.org/jira/browse/SPARK-61188): raise 
`RecursionError` instead of crashing the process when progress handlers and 
hooks run queries without end.
    
    - `spark_connect_core::runtime` counts the progress handlers and the hooks 
that run on a thread (`CallbackGuard`). `check_callback_depth()` fails an 
execution that starts past `MAX_CALLBACK_DEPTH` (2) nested callbacks with a new 
`SparkErrorKind::RecursionError`. After such a failure, the callbacks above it 
fail their queries at once until the outermost callback returns, so the 
failures do not multiply.
    - The check is made when a dataframe execution starts 
(`SparkSession::execute_plan`) and before a hook is called 
(`SparkConnectClient::hooked`). The progress handlers and the hooks are called 
inside a guard.
    - The Python layer raises the error of that kind as the builtin 
`RecursionError`. Its exceptions in handlers and hooks keep their existing 
behavior: a handler's exception goes to `sys.unraisablehook`, a hook's 
exception is raised from the execution that calls it.
    
    Official PySpark 4.3.0 with the same unguarded handler: the outer 
`collect()` raises `SparkConnectGrpcException` ("Exception serializing 
request!"), since the handler's exception is raised out of the progress loop of 
the outer query, and the process survives. This change keeps the process alive 
and returns the outer result. The exception of a nested query is printed by 
`sys.unraisablehook` (`RecursionError`), because a progress handler's exception 
does not fail the execution in this c [...]
    
    ### Why are the changes needed?
    
    ```python
    spark.registerProgressHandler(lambda **kw: spark.range(4).count() if not 
kw["done"] else None)
    spark.range(2000000).selectExpr("sum(id) as t").collect()
    # exit 139 (SIGSEGV), no Python exception
    ```
    
    Each nested query runs the handler again, and each level takes native 
stack, so the stack of the process overflows before Python's recursion limit is 
reached. Measured: about 100 KiB per level, so four levels crash a 512 KiB 
thread and about 20 crash a 2 MiB thread. The bound of two levels leaves room 
for the frames of the thread itself. Guarded handlers and hooks that run one 
nested query are unaffected.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. An unguarded progress handler or hook that runs queries without end no 
longer crashes the process: the query past the bound raises `RecursionError` in 
the callback, and the outer query returns its result.
    
    ### How was this patch tested?
    
    - Live 
`test_progress_handler_that_runs_queries_without_end_raises_recursion_error` 
runs the unguarded handler above in a subprocess: the return code is 0, the 
outer result is `[Row(t=1999999000000)]`, and stderr has the `RecursionError`. 
Without the bound the subprocess dies with -11.
    - Offline `test_progress_handler_that_runs_queries_without_end_is_bounded` 
runs the same handler on the mock Connect service: the handler is called for 
two nested levels, one `RecursionError` goes to `sys.unraisablehook`, and the 
outer query returns. Without the bound the test process segfaults.
    - Rust `test_callback_depth_is_bounded_and_released` in `runtime.rs`: the 
depth check fails at the bound and is released when the callbacks end.
    - The existing live 
`test_progress_handler_runs_a_query_of_its_own_as_pyspark` and 
`test_hook_runs_a_query_of_its_own_as_pyspark` still pass (one nested query).
    - The tests of both crates, e2e included, against a local Spark 4.3.0 
server (895 passed), the build of all tests, also with `wasm-udf`, the offline 
Python suites (2228 passed, 8 skipped), and the live drop-in, Arrow and ML 
suites (210 passed) and live cases (10 passed); after moving onto master with 
SPARK-61186, the lib tests of both crates and the progress, hook and tag tests 
again. rustfmt passes, and clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #368 from HyukjinKwon/SPARK-61188-bounded-callback-recursion.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/errors.rs          |  5 +++
 crates/spark-connect-core/src/client.rs  |  3 ++
 crates/spark-connect-core/src/error.rs   | 19 ++++++++++
 crates/spark-connect-core/src/runtime.rs | 65 ++++++++++++++++++++++++++++++++
 crates/spark-connect/src/session.rs      |  5 ++-
 python/tests/test_dropin_e2e.py          | 23 +++++++++++
 python/tests/test_dropin_offline.py      | 30 +++++++++++++++
 7 files changed, 149 insertions(+), 1 deletion(-)

diff --git a/crates/pyspark-rs/src/errors.rs b/crates/pyspark-rs/src/errors.rs
index b9a17f7c..2b780b49 100644
--- a/crates/pyspark-rs/src/errors.rs
+++ b/crates/pyspark-rs/src/errors.rs
@@ -16,6 +16,7 @@ fn kind_to_pyspark_class_name(kind: SparkErrorKind) -> 
&'static str {
         SparkErrorKind::AttributeError => "PySparkAttributeError",
         SparkErrorKind::KeyError => "PySparkKeyError",
         SparkErrorKind::RuntimeError => "PySparkRuntimeError",
+        SparkErrorKind::RecursionError => "RecursionError",
         SparkErrorKind::NotImplementedError => "PySparkNotImplementedError",
         SparkErrorKind::AssertionError => "PySparkAssertionError",
         SparkErrorKind::PicklingError => "PySparkPicklingError",
@@ -190,6 +191,10 @@ pub fn spark_error_to_py_exception(err: SparkError) -> 
PyErr {
     if let Some(exception) = callback_exception(&err) {
         return exception;
     }
+    if err.kind == SparkErrorKind::RecursionError {
+        // A builtin, as PySpark's recursion limit raises it.
+        return pyo3::exceptions::PyRecursionError::new_err(err.message());
+    }
     let msg = err.message();
     let operation_id = err.operation_id.clone();
     let class_name = kind_to_pyspark_class_name(err.kind);
diff --git a/crates/spark-connect-core/src/client.rs 
b/crates/spark-connect-core/src/client.rs
index a01eeb7b..aedbec9e 100644
--- a/crates/spark-connect-core/src/client.rs
+++ b/crates/spark-connect-core/src/client.rs
@@ -437,7 +437,10 @@ impl SparkConnectClient {
         let Some(hook) = hook else {
             return Ok(request);
         };
+        crate::runtime::check_callback_depth()?;
         let operation_id = request.operation_id.clone();
+        // The hook may run a query of its own, which runs the hooks too.
+        let _callback = crate::runtime::CallbackGuard::enter();
         let mut request = hook(request)?;
         request.operation_id = operation_id;
         Ok(request)
diff --git a/crates/spark-connect-core/src/error.rs 
b/crates/spark-connect-core/src/error.rs
index 791813ab..ebf6baa6 100644
--- a/crates/spark-connect-core/src/error.rs
+++ b/crates/spark-connect-core/src/error.rs
@@ -273,6 +273,8 @@ pub enum SparkErrorKind {
     QueryExecution,
     /// ``pyspark.errors.exceptions.base.PythonException``
     Python,
+    /// ``RecursionError``, a Python builtin: the executions nested in 
callbacks are too deep.
+    RecursionError,
     /// ``pyspark.errors.exceptions.base.SparkRuntimeException``
     SparkRuntime,
     /// ``pyspark.errors.exceptions.base.SparkUpgradeException``
@@ -334,6 +336,23 @@ impl SparkError {
         }
     }
 
+    /// Create a RecursionError-kind error with a plain message (no error 
class).
+    pub fn recursion_msg(message: impl Into<String>) -> Self {
+        Self {
+            kind: SparkErrorKind::RecursionError,
+            error_class: String::new(),
+            params: BTreeMap::new(),
+            message: message.into(),
+            sql_state: None,
+            contexts: Vec::new(),
+            server_stacktrace: None,
+            grpc_code: None,
+            grpc_details: None,
+            operation_id: None,
+            server: None,
+        }
+    }
+
     /// Create an error of a specific kind with an error class and parameters.
     pub fn classed(kind: SparkErrorKind, error_class: &str, params: &[(&str, 
&str)]) -> Self {
         Self {
diff --git a/crates/spark-connect-core/src/runtime.rs 
b/crates/spark-connect-core/src/runtime.rs
index b694f281..7d6b25ed 100644
--- a/crates/spark-connect-core/src/runtime.rs
+++ b/crates/spark-connect-core/src/runtime.rs
@@ -42,10 +42,75 @@ pub fn block_on<F: std::future::Future>(f: F) -> F::Output {
     }
 }
 
+/// How many progress handlers and hooks may run nested on a thread, each one 
running a query
+/// that runs the next one. Past it, an execution fails with a 
`RecursionError` (as PySpark's
+/// recursion limit raises it) instead of recursing until the native stack of 
the thread
+/// overflows. A level takes about 100 KiB of native stack (Python frames of 
the handler, and
+/// the calls of PyO3 between them): on a thread of 512 KiB, four levels 
crash, so two levels
+/// leave room for the frames of the thread itself.
+pub const MAX_CALLBACK_DEPTH: usize = 2;
+
+thread_local! {
+    static CALLBACK_DEPTH: std::cell::Cell<usize> = const { 
std::cell::Cell::new(0) };
+    /// Set once an execution failed past the bound, until the outermost 
callback returns: the
+    /// callbacks of the levels above then fail their queries at once, rather 
than each level
+    /// running a query per progress update again, which would grow 
exponentially.
+    static CALLBACK_RECURSION_FAILED: std::cell::Cell<bool> = const { 
std::cell::Cell::new(false) };
+}
+
+/// A progress handler or a hook that runs on this thread, for as long as it 
lives.
+pub struct CallbackGuard(());
+
+impl CallbackGuard {
+    /// Enter a callback that may run a query of its own.
+    pub fn enter() -> Self {
+        CALLBACK_DEPTH.with(|depth| depth.set(depth.get() + 1));
+        CallbackGuard(())
+    }
+}
+
+impl Drop for CallbackGuard {
+    fn drop(&mut self) {
+        let depth = CALLBACK_DEPTH.with(|depth| {
+            depth.set(depth.get() - 1);
+            depth.get()
+        });
+        if depth == 0 {
+            CALLBACK_RECURSION_FAILED.with(|failed| failed.set(false));
+        }
+    }
+}
+
+/// Fail with a `RecursionError` when the executions started from this thread 
are nested past
+/// [`MAX_CALLBACK_DEPTH`] callbacks, instead of the stack of the thread 
running out.
+#[allow(clippy::result_large_err)] // SparkError is large, as in the other 
entry points
+pub fn check_callback_depth() -> crate::error::Result<()> {
+    let too_deep = CALLBACK_DEPTH.with(|depth| depth.get()) >= 
MAX_CALLBACK_DEPTH;
+    if too_deep || CALLBACK_RECURSION_FAILED.with(|failed| failed.get()) {
+        CALLBACK_RECURSION_FAILED.with(|failed| failed.set(true));
+        return Err(crate::error::SparkError::recursion_msg(
+            "maximum recursion depth exceeded in a progress handler or a hook 
of an execution",
+        ));
+    }
+    Ok(())
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
 
+    #[test]
+    fn test_callback_depth_is_bounded_and_released() {
+        assert!(check_callback_depth().is_ok());
+        let guards: Vec<CallbackGuard> = (0..MAX_CALLBACK_DEPTH)
+            .map(|_| CallbackGuard::enter())
+            .collect();
+        let err = check_callback_depth().unwrap_err();
+        assert_eq!(err.kind, crate::error::SparkErrorKind::RecursionError);
+        drop(guards);
+        assert!(check_callback_depth().is_ok());
+    }
+
     #[test]
     fn test_runtime_created_once() {
         let rt1 = get_runtime();
diff --git a/crates/spark-connect/src/session.rs 
b/crates/spark-connect/src/session.rs
index caf94421..6dc8d152 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -8,7 +8,7 @@ use std::sync::{Arc, Mutex};
 use spark_connect_core::channel::ChannelBuilder;
 use spark_connect_core::client::{ExecutePlanHook, SparkConnectClient};
 use spark_connect_core::error::{Result, SparkError};
-use spark_connect_core::runtime::block_on;
+use spark_connect_core::runtime::{block_on, check_callback_depth, 
CallbackGuard};
 use spark_connect_core::thread_entries::forget_when_thread_ends;
 use spark_connect_proto as proto;
 
@@ -286,6 +286,8 @@ impl SparkSession {
             return;
         }
         let update = update();
+        // A handler may run a query of its own, whose progress calls the 
handlers again.
+        let _callback = CallbackGuard::enter();
         for h in handlers {
             h(&update);
         }
@@ -303,6 +305,7 @@ impl SparkSession {
             .get_or_insert_with(|| uuid::Uuid::new_v4().to_string())
             .clone();
         // Before the request: an execution that fails to start ends, as done, 
too.
+        check_callback_depth()?;
         let progress = ProgressTracker::new(self, Some(operation_id));
         let stream = 
block_on(self.client().execute_plan_reattachable(request))?;
         Ok(ExecutionStream::new(stream, progress))
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 157284eb..aea3b595 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -984,6 +984,29 @@ def 
test_foreach_foreach_partition_and_to_json_as_pyspark(session):
     assert json.collect() == [Row(value='{"id":1,"name":"a","bin":"eA=="}'), 
Row(value='{"id":2}')]
 
 
+def 
test_progress_handler_that_runs_queries_without_end_raises_recursion_error():
+    # The unguarded handler of SPARK-61153 runs a query for each progress of 
the queries it
+    # runs. Past the bound of nested callbacks, the query fails with a 
RecursionError, which the
+    # handler's exception goes out with, instead of the native stack 
overflowing and the process
+    # crashing (SIGSEGV). The outer query returns its result, as the handler's 
exceptions do.
+    import subprocess
+    import sys
+
+    remote = os.environ.get("SPARK_REMOTE")
+    if not remote:
+        pytest.skip("SPARK_REMOTE is required for live-server tests")
+    result = subprocess.run([sys.executable, "-c", """
+import sys
+from pyspark.sql import SparkSession
+spark = SparkSession.builder.remote(%r).getOrCreate()
+spark.registerProgressHandler(lambda **kw: spark.range(4).count() if not 
kw["done"] else None)
+print(spark.range(2000000).selectExpr("sum(id) as t").collect())
+""" % remote], env=dict(os.environ, TZ="UTC"), capture_output=True, text=True, 
timeout=300)
+    assert result.returncode == 0, result.stderr
+    assert "[Row(t=1999999000000)]" in result.stdout
+    assert "RecursionError" in result.stderr
+
+
 def test_progress_handler_runs_a_query_of_its_own_as_pyspark(session):
     # As PySpark's: a handler runs while the execution runs, and a query of 
the handler runs
     # too, the progress of the execution having no effect on it.
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index abdebcb8..3c7c9695 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -8579,6 +8579,36 @@ def 
test_udf_profiles_are_collected_from_the_python_accumulators_as_pyspark(
 
 
 
+def 
test_progress_handler_that_runs_queries_without_end_is_bounded(mock_connect, 
monkeypatch):
+    # A handler that runs a query for each progress of an execution nests the 
executions past
+    # the bound of nested callbacks: the query past it fails with a 
RecursionError, which the
+    # handler's exception goes to sys.unraisablehook with, instead of the 
native stack
+    # overflowing, and the outer query returns.
+    import sys
+
+    spark, service = mock_connect
+    # ExecutePlanResponse.execution_progress=18 -> one stage that is not done.
+    stage = (varint_field(1, 7) + varint_field(2, 4) + varint_field(3, 1)
+             + varint_field(4, 1024) + varint_field(5, 0))
+    service.execute_payload = message_field(18, message_field(1, stage) + 
varint_field(2, 3))
+    unraisable = []
+    monkeypatch.setattr(sys, "unraisablehook", unraisable.append)
+    calls = []
+
+    def handler(**kwargs):
+        calls.append(kwargs["done"])
+        if not kwargs["done"]:
+            spark.range(1).collect()
+
+    spark.registerProgressHandler(handler)
+    try:
+        spark.range(1).collect()
+    finally:
+        spark.removeProgressHandler(handler)
+    assert calls == [False, False, True, True]
+    assert [type(error.exc_value) for error in unraisable] == [RecursionError]
+
+
 def test_schema_cache_cold_children_keep_independent_caches(mock_connect):
     spark, service = mock_connect
     df = spark.range(10)


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to