This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new bb1f5a19 [SPARK-61188][CONNECT][PYTHON] Raise RecursionError instead
of crashing when progress handlers run queries without end, in the Python
client as PySpark does
bb1f5a19 is described below
commit bb1f5a1931e7e37335046a071edb55c048c3d917
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 19:41:08 2026 +0900
[SPARK-61188][CONNECT][PYTHON] Raise RecursionError instead of crashing
when progress handlers run queries without end, in the Python client as PySpark
does
### What changes were proposed in this pull request?
[SPARK-61188](https://issues.apache.org/jira/browse/SPARK-61188): raise
`RecursionError` instead of crashing the process when progress handlers and
hooks run queries without end.
- `spark_connect_core::runtime` counts the progress handlers and the hooks
that run on a thread (`CallbackGuard`). `check_callback_depth()` fails an
execution that starts past `MAX_CALLBACK_DEPTH` (2) nested callbacks with a new
`SparkErrorKind::RecursionError`. After such a failure, the callbacks above it
fail their queries at once until the outermost callback returns, so the
failures do not multiply.
- The check is made when a dataframe execution starts
(`SparkSession::execute_plan`) and before a hook is called
(`SparkConnectClient::hooked`). The progress handlers and the hooks are called
inside a guard.
- The Python layer raises the error of that kind as the builtin
`RecursionError`. Its exceptions in handlers and hooks keep their existing
behavior: a handler's exception goes to `sys.unraisablehook`, a hook's
exception is raised from the execution that calls it.
Official PySpark 4.3.0 with the same unguarded handler: the outer
`collect()` raises `SparkConnectGrpcException` ("Exception serializing
request!"), since the handler's exception is raised out of the progress loop of
the outer query, and the process survives. This change keeps the process alive
and returns the outer result. The exception of a nested query is printed by
`sys.unraisablehook` (`RecursionError`), because a progress handler's exception
does not fail the execution in this c [...]
### Why are the changes needed?
```python
spark.registerProgressHandler(lambda **kw: spark.range(4).count() if not
kw["done"] else None)
spark.range(2000000).selectExpr("sum(id) as t").collect()
# exit 139 (SIGSEGV), no Python exception
```
Each nested query runs the handler again, and each level takes native
stack, so the stack of the process overflows before Python's recursion limit is
reached. Measured: about 100 KiB per level, so four levels crash a 512 KiB
thread and about 20 crash a 2 MiB thread. The bound of two levels leaves room
for the frames of the thread itself. Guarded handlers and hooks that run one
nested query are unaffected.
### Does this PR introduce _any_ user-facing change?
Yes. An unguarded progress handler or hook that runs queries without end no
longer crashes the process: the query past the bound raises `RecursionError` in
the callback, and the outer query returns its result.
### How was this patch tested?
- Live
`test_progress_handler_that_runs_queries_without_end_raises_recursion_error`
runs the unguarded handler above in a subprocess: the return code is 0, the
outer result is `[Row(t=1999999000000)]`, and stderr has the `RecursionError`.
Without the bound the subprocess dies with -11.
- Offline `test_progress_handler_that_runs_queries_without_end_is_bounded`
runs the same handler on the mock Connect service: the handler is called for
two nested levels, one `RecursionError` goes to `sys.unraisablehook`, and the
outer query returns. Without the bound the test process segfaults.
- Rust `test_callback_depth_is_bounded_and_released` in `runtime.rs`: the
depth check fails at the bound and is released when the callbacks end.
- The existing live
`test_progress_handler_runs_a_query_of_its_own_as_pyspark` and
`test_hook_runs_a_query_of_its_own_as_pyspark` still pass (one nested query).
- The tests of both crates, e2e included, against a local Spark 4.3.0
server (895 passed), the build of all tests, also with `wasm-udf`, the offline
Python suites (2228 passed, 8 skipped), and the live drop-in, Arrow and ML
suites (210 passed) and live cases (10 passed); after moving onto master with
SPARK-61186, the lib tests of both crates and the progress, hook and tag tests
again. rustfmt passes, and clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #368 from HyukjinKwon/SPARK-61188-bounded-callback-recursion.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/errors.rs | 5 +++
crates/spark-connect-core/src/client.rs | 3 ++
crates/spark-connect-core/src/error.rs | 19 ++++++++++
crates/spark-connect-core/src/runtime.rs | 65 ++++++++++++++++++++++++++++++++
crates/spark-connect/src/session.rs | 5 ++-
python/tests/test_dropin_e2e.py | 23 +++++++++++
python/tests/test_dropin_offline.py | 30 +++++++++++++++
7 files changed, 149 insertions(+), 1 deletion(-)
diff --git a/crates/pyspark-rs/src/errors.rs b/crates/pyspark-rs/src/errors.rs
index b9a17f7c..2b780b49 100644
--- a/crates/pyspark-rs/src/errors.rs
+++ b/crates/pyspark-rs/src/errors.rs
@@ -16,6 +16,7 @@ fn kind_to_pyspark_class_name(kind: SparkErrorKind) ->
&'static str {
SparkErrorKind::AttributeError => "PySparkAttributeError",
SparkErrorKind::KeyError => "PySparkKeyError",
SparkErrorKind::RuntimeError => "PySparkRuntimeError",
+ SparkErrorKind::RecursionError => "RecursionError",
SparkErrorKind::NotImplementedError => "PySparkNotImplementedError",
SparkErrorKind::AssertionError => "PySparkAssertionError",
SparkErrorKind::PicklingError => "PySparkPicklingError",
@@ -190,6 +191,10 @@ pub fn spark_error_to_py_exception(err: SparkError) ->
PyErr {
if let Some(exception) = callback_exception(&err) {
return exception;
}
+ if err.kind == SparkErrorKind::RecursionError {
+ // A builtin, as PySpark's recursion limit raises it.
+ return pyo3::exceptions::PyRecursionError::new_err(err.message());
+ }
let msg = err.message();
let operation_id = err.operation_id.clone();
let class_name = kind_to_pyspark_class_name(err.kind);
diff --git a/crates/spark-connect-core/src/client.rs
b/crates/spark-connect-core/src/client.rs
index a01eeb7b..aedbec9e 100644
--- a/crates/spark-connect-core/src/client.rs
+++ b/crates/spark-connect-core/src/client.rs
@@ -437,7 +437,10 @@ impl SparkConnectClient {
let Some(hook) = hook else {
return Ok(request);
};
+ crate::runtime::check_callback_depth()?;
let operation_id = request.operation_id.clone();
+ // The hook may run a query of its own, which runs the hooks too.
+ let _callback = crate::runtime::CallbackGuard::enter();
let mut request = hook(request)?;
request.operation_id = operation_id;
Ok(request)
diff --git a/crates/spark-connect-core/src/error.rs
b/crates/spark-connect-core/src/error.rs
index 791813ab..ebf6baa6 100644
--- a/crates/spark-connect-core/src/error.rs
+++ b/crates/spark-connect-core/src/error.rs
@@ -273,6 +273,8 @@ pub enum SparkErrorKind {
QueryExecution,
/// ``pyspark.errors.exceptions.base.PythonException``
Python,
+ /// ``RecursionError``, a Python builtin: the executions nested in
callbacks are too deep.
+ RecursionError,
/// ``pyspark.errors.exceptions.base.SparkRuntimeException``
SparkRuntime,
/// ``pyspark.errors.exceptions.base.SparkUpgradeException``
@@ -334,6 +336,23 @@ impl SparkError {
}
}
+ /// Create a RecursionError-kind error with a plain message (no error
class).
+ pub fn recursion_msg(message: impl Into<String>) -> Self {
+ Self {
+ kind: SparkErrorKind::RecursionError,
+ error_class: String::new(),
+ params: BTreeMap::new(),
+ message: message.into(),
+ sql_state: None,
+ contexts: Vec::new(),
+ server_stacktrace: None,
+ grpc_code: None,
+ grpc_details: None,
+ operation_id: None,
+ server: None,
+ }
+ }
+
/// Create an error of a specific kind with an error class and parameters.
pub fn classed(kind: SparkErrorKind, error_class: &str, params: &[(&str,
&str)]) -> Self {
Self {
diff --git a/crates/spark-connect-core/src/runtime.rs
b/crates/spark-connect-core/src/runtime.rs
index b694f281..7d6b25ed 100644
--- a/crates/spark-connect-core/src/runtime.rs
+++ b/crates/spark-connect-core/src/runtime.rs
@@ -42,10 +42,75 @@ pub fn block_on<F: std::future::Future>(f: F) -> F::Output {
}
}
+/// How many progress handlers and hooks may run nested on a thread, each one
running a query
+/// that runs the next one. Past it, an execution fails with a
`RecursionError` (as PySpark's
+/// recursion limit raises it) instead of recursing until the native stack of
the thread
+/// overflows. A level takes about 100 KiB of native stack (Python frames of
the handler, and
+/// the calls of PyO3 between them): on a thread of 512 KiB, four levels
crash, so two levels
+/// leave room for the frames of the thread itself.
+pub const MAX_CALLBACK_DEPTH: usize = 2;
+
+thread_local! {
+ static CALLBACK_DEPTH: std::cell::Cell<usize> = const {
std::cell::Cell::new(0) };
+ /// Set once an execution failed past the bound, until the outermost
callback returns: the
+ /// callbacks of the levels above then fail their queries at once, rather
than each level
+ /// running a query per progress update again, which would grow
exponentially.
+ static CALLBACK_RECURSION_FAILED: std::cell::Cell<bool> = const {
std::cell::Cell::new(false) };
+}
+
+/// A progress handler or a hook that runs on this thread, for as long as it
lives.
+pub struct CallbackGuard(());
+
+impl CallbackGuard {
+ /// Enter a callback that may run a query of its own.
+ pub fn enter() -> Self {
+ CALLBACK_DEPTH.with(|depth| depth.set(depth.get() + 1));
+ CallbackGuard(())
+ }
+}
+
+impl Drop for CallbackGuard {
+ fn drop(&mut self) {
+ let depth = CALLBACK_DEPTH.with(|depth| {
+ depth.set(depth.get() - 1);
+ depth.get()
+ });
+ if depth == 0 {
+ CALLBACK_RECURSION_FAILED.with(|failed| failed.set(false));
+ }
+ }
+}
+
+/// Fail with a `RecursionError` when the executions started from this thread
are nested past
+/// [`MAX_CALLBACK_DEPTH`] callbacks, instead of the stack of the thread
running out.
+#[allow(clippy::result_large_err)] // SparkError is large, as in the other
entry points
+pub fn check_callback_depth() -> crate::error::Result<()> {
+ let too_deep = CALLBACK_DEPTH.with(|depth| depth.get()) >=
MAX_CALLBACK_DEPTH;
+ if too_deep || CALLBACK_RECURSION_FAILED.with(|failed| failed.get()) {
+ CALLBACK_RECURSION_FAILED.with(|failed| failed.set(true));
+ return Err(crate::error::SparkError::recursion_msg(
+ "maximum recursion depth exceeded in a progress handler or a hook
of an execution",
+ ));
+ }
+ Ok(())
+}
+
#[cfg(test)]
mod tests {
use super::*;
+ #[test]
+ fn test_callback_depth_is_bounded_and_released() {
+ assert!(check_callback_depth().is_ok());
+ let guards: Vec<CallbackGuard> = (0..MAX_CALLBACK_DEPTH)
+ .map(|_| CallbackGuard::enter())
+ .collect();
+ let err = check_callback_depth().unwrap_err();
+ assert_eq!(err.kind, crate::error::SparkErrorKind::RecursionError);
+ drop(guards);
+ assert!(check_callback_depth().is_ok());
+ }
+
#[test]
fn test_runtime_created_once() {
let rt1 = get_runtime();
diff --git a/crates/spark-connect/src/session.rs
b/crates/spark-connect/src/session.rs
index caf94421..6dc8d152 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -8,7 +8,7 @@ use std::sync::{Arc, Mutex};
use spark_connect_core::channel::ChannelBuilder;
use spark_connect_core::client::{ExecutePlanHook, SparkConnectClient};
use spark_connect_core::error::{Result, SparkError};
-use spark_connect_core::runtime::block_on;
+use spark_connect_core::runtime::{block_on, check_callback_depth,
CallbackGuard};
use spark_connect_core::thread_entries::forget_when_thread_ends;
use spark_connect_proto as proto;
@@ -286,6 +286,8 @@ impl SparkSession {
return;
}
let update = update();
+ // A handler may run a query of its own, whose progress calls the
handlers again.
+ let _callback = CallbackGuard::enter();
for h in handlers {
h(&update);
}
@@ -303,6 +305,7 @@ impl SparkSession {
.get_or_insert_with(|| uuid::Uuid::new_v4().to_string())
.clone();
// Before the request: an execution that fails to start ends, as done,
too.
+ check_callback_depth()?;
let progress = ProgressTracker::new(self, Some(operation_id));
let stream =
block_on(self.client().execute_plan_reattachable(request))?;
Ok(ExecutionStream::new(stream, progress))
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 157284eb..aea3b595 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -984,6 +984,29 @@ def
test_foreach_foreach_partition_and_to_json_as_pyspark(session):
assert json.collect() == [Row(value='{"id":1,"name":"a","bin":"eA=="}'),
Row(value='{"id":2}')]
+def
test_progress_handler_that_runs_queries_without_end_raises_recursion_error():
+ # The unguarded handler of SPARK-61153 runs a query for each progress of
the queries it
+ # runs. Past the bound of nested callbacks, the query fails with a
RecursionError, which the
+ # handler's exception goes out with, instead of the native stack
overflowing and the process
+ # crashing (SIGSEGV). The outer query returns its result, as the handler's
exceptions do.
+ import subprocess
+ import sys
+
+ remote = os.environ.get("SPARK_REMOTE")
+ if not remote:
+ pytest.skip("SPARK_REMOTE is required for live-server tests")
+ result = subprocess.run([sys.executable, "-c", """
+import sys
+from pyspark.sql import SparkSession
+spark = SparkSession.builder.remote(%r).getOrCreate()
+spark.registerProgressHandler(lambda **kw: spark.range(4).count() if not
kw["done"] else None)
+print(spark.range(2000000).selectExpr("sum(id) as t").collect())
+""" % remote], env=dict(os.environ, TZ="UTC"), capture_output=True, text=True,
timeout=300)
+ assert result.returncode == 0, result.stderr
+ assert "[Row(t=1999999000000)]" in result.stdout
+ assert "RecursionError" in result.stderr
+
+
def test_progress_handler_runs_a_query_of_its_own_as_pyspark(session):
# As PySpark's: a handler runs while the execution runs, and a query of
the handler runs
# too, the progress of the execution having no effect on it.
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index abdebcb8..3c7c9695 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -8579,6 +8579,36 @@ def
test_udf_profiles_are_collected_from_the_python_accumulators_as_pyspark(
+def
test_progress_handler_that_runs_queries_without_end_is_bounded(mock_connect,
monkeypatch):
+ # A handler that runs a query for each progress of an execution nests the
executions past
+ # the bound of nested callbacks: the query past it fails with a
RecursionError, which the
+ # handler's exception goes to sys.unraisablehook with, instead of the
native stack
+ # overflowing, and the outer query returns.
+ import sys
+
+ spark, service = mock_connect
+ # ExecutePlanResponse.execution_progress=18 -> one stage that is not done.
+ stage = (varint_field(1, 7) + varint_field(2, 4) + varint_field(3, 1)
+ + varint_field(4, 1024) + varint_field(5, 0))
+ service.execute_payload = message_field(18, message_field(1, stage) +
varint_field(2, 3))
+ unraisable = []
+ monkeypatch.setattr(sys, "unraisablehook", unraisable.append)
+ calls = []
+
+ def handler(**kwargs):
+ calls.append(kwargs["done"])
+ if not kwargs["done"]:
+ spark.range(1).collect()
+
+ spark.registerProgressHandler(handler)
+ try:
+ spark.range(1).collect()
+ finally:
+ spark.removeProgressHandler(handler)
+ assert calls == [False, False, True, True]
+ assert [type(error.exc_value) for error in unraisable] == [RecursionError]
+
+
def test_schema_cache_cold_children_keep_independent_caches(mock_connect):
spark, service = mock_connect
df = spark.range(10)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]