This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 2b89466f [SPARK-61153][CONNECT][PYTHON] Run the queries of progress
handlers and hooks, and keep an execution reattachable when a hook adds
reattach options, as PySpark does
2b89466f is described below
commit 2b89466f97855cae890a9fe96986f2082a60430f
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 12:47:02 2026 +0900
[SPARK-61153][CONNECT][PYTHON] Run the queries of progress handlers and
hooks, and keep an execution reattachable when a hook adds reattach options, as
PySpark does
### What changes were proposed in this pull request?
[SPARK-61153](https://issues.apache.org/jira/browse/SPARK-61153): a query
run inside a progress handler or a `SparkSession.Hook` runs, and a hook that
adds `reattachable = False` keeps the execution reattachable, as PySpark does.
- `spark_connect_core::runtime::block_on` blocks in place when it is called
from a thread that is already in the shared multi-thread runtime's context,
which is the case for a callback that runs inside an execution. The nested
query then runs, as it does in PySpark. On a `current_thread` runtime the call
still fails as before, since such a runtime cannot block in place. No lock is
held across the callback: the progress handlers and the hook are cloned out of
their locks before they ar [...]
- `ExecutePlanResponseReattachableIterator::new` always appends its
`reattachable = True` option, after the request options that the hooks added,
as PySpark's `ExecutePlanResponseReattachableIterator` does. Before, it
appended the option only when the request had none, so a hook's `reattachable =
False` made the server treat the execution as not reattachable, and the rows
were lost with `INVALID_HANDLE.OPERATION_NOT_FOUND`.
- Two unit tests that relied on the nested-runtime panic now use a
`current_thread` runtime, which keeps the panic they check: the schema cache of
a dataframe and the version cache of a session.
### Why are the changes needed?
Both defects run Python callbacks while an execution runs:
```python
s.registerProgressHandler(lambda **kw: s.range(4).count() if not kw["done"]
else None)
s.range(2000000).selectExpr("sum(id) as t").collect()
# PanicException: Cannot start a runtime from within a runtime
class H(SparkSession.Hook):
def on_execute_plan(self, request):
other.range(2).count() # the same panic
return request
b._registerHook(lambda s: H())
b._registerHook(lambda s: AppendNotReattachable()) # appends
request_options reattach_options.reattachable = False
s.range(2).count()
# INVALID_HANDLE.OPERATION_NOT_FOUND after the rows are read
```
Official PySpark 4.3.0 gives `nested count in handler -> 4`, `outer ->
[Row(t=1999999000000)]`, `outer hook query -> 3`, and `(2, [Row(x=1)])` for the
last case.
### Does this PR introduce _any_ user-facing change?
Yes. A query run in a progress handler or a session hook returns its result
instead of panicking, and a hook that appends a non-reattachable option no
longer fails the execution.
### How was this patch tested?
- Rust `test_block_on_nested_in_block_on` and
`test_block_on_nested_on_a_worker_thread` in `runtime.rs`: a `block_on` from
within `block_on`, and from a worker thread.
- Rust `test_reattach_options_set_after_a_hook_added_a_false_one` in
`reattach.rs`: the iterator's options are `[false, true]` after a hook added
`false`.
- Live `test_progress_handler_runs_a_query_of_its_own_as_pyspark`: a
handler runs a count while the execution runs. The same results with official
PySpark 4.3.0.
- Live `test_hook_runs_a_query_of_its_own_as_pyspark`: a hook runs a query.
The same results with official PySpark 4.3.0.
- Live
`test_hook_appending_a_non_reattachable_option_keeps_the_execution_reattachable`:
a count and a `sql` collect with a hook that appends `reattachable = False`.
The same results with official PySpark 4.3.0.
- The tests of both crates, e2e included, against a local Spark 4.3.0
server (878 passed), the build of all tests, also with `wasm-udf`, the offline
Python suites (2201 passed, 8 skipped), and the live drop-in, Arrow and ML
suites (187 passed) and live cases (10 passed). rustfmt passes, and clippy
reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #342 from HyukjinKwon/SPARK-61153-nested-queries-hooks-reattach.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/spark-connect-core/src/reattach.rs | 58 ++++++++++++++++++-------
crates/spark-connect-core/src/runtime.rs | 27 +++++++++++-
crates/spark-connect/src/dataframe.rs | 6 ++-
crates/spark-connect/src/session.rs | 10 +++--
python/tests/test_dropin_e2e.py | 70 +++++++++++++++++++++++++++++++
5 files changed, 152 insertions(+), 19 deletions(-)
diff --git a/crates/spark-connect-core/src/reattach.rs
b/crates/spark-connect-core/src/reattach.rs
index d5c20281..e21feb38 100644
--- a/crates/spark-connect-core/src/reattach.rs
+++ b/crates/spark-connect-core/src/reattach.rs
@@ -43,20 +43,18 @@ impl ExecutePlanResponseReattachableIterator {
request.operation_id = Some(operation_id.clone());
- // Ensure reattachable options are set
- if !request.request_options.iter().any(|opt| {
- matches!(
- &opt.request_option,
-
Some(spark_connect_proto::execute_plan_request::request_option::RequestOption::ReattachOptions(_))
- )
- }) {
- let mut req_opt =
spark_connect_proto::execute_plan_request::RequestOption::default();
- req_opt.request_option =
-
Some(spark_connect_proto::execute_plan_request::request_option::RequestOption::ReattachOptions(
- spark_connect_proto::ReattachOptions { reattachable: true
},
- ));
- request.request_options.push(req_opt);
- }
+ // Always append the reattachable option, as PySpark does: a hook may
have added one
+ // with `reattachable = False`, which the server takes as not
reattachable only when no
+ // option says true.
+ request
+ .request_options
+ .push(spark_connect_proto::execute_plan_request::RequestOption {
+ request_option: Some(
+
spark_connect_proto::execute_plan_request::request_option::RequestOption::ReattachOptions(
+ spark_connect_proto::ReattachOptions { reattachable:
true },
+ ),
+ ),
+ });
Self {
operation_id,
@@ -245,6 +243,38 @@ mod tests {
}));
}
+ #[tokio::test]
+ async fn test_reattach_options_set_after_a_hook_added_a_false_one() {
+ // A session hook appends `reattachable = False`; the execution still
ends up with an
+ // option that says true, as PySpark's iterator appends its own after
the hooks ran.
+ let request = ExecutePlanRequest {
+ session_id: "session-123".to_string(),
+ request_options:
vec![spark_connect_proto::execute_plan_request::RequestOption {
+ request_option: Some(
+
spark_connect_proto::execute_plan_request::request_option::RequestOption::ReattachOptions(
+ spark_connect_proto::ReattachOptions {
+ reattachable: false,
+ },
+ ),
+ ),
+ }],
+ ..Default::default()
+ };
+
+ let iter = ExecutePlanResponseReattachableIterator::new(request);
+
+ let reattachable: Vec<bool> = iter
+ .request()
+ .request_options
+ .iter()
+ .filter_map(|opt| match &opt.request_option {
+
Some(spark_connect_proto::execute_plan_request::request_option::RequestOption::ReattachOptions(ro))
=> Some(ro.reattachable),
+ _ => None,
+ })
+ .collect();
+ assert_eq!(reattachable, vec![false, true]);
+ }
+
#[tokio::test]
async fn test_create_reattach_request() {
let mut request = ExecutePlanRequest::default();
diff --git a/crates/spark-connect-core/src/runtime.rs
b/crates/spark-connect-core/src/runtime.rs
index 070c6f5c..b694f281 100644
--- a/crates/spark-connect-core/src/runtime.rs
+++ b/crates/spark-connect-core/src/runtime.rs
@@ -27,8 +27,19 @@ pub fn get_runtime() -> &'static Runtime {
/// Allows synchronous code (e.g., from PyO3) to execute an async function
/// and wait for its result. This should be called with the GIL released
/// in a Python context.
+///
+/// Also called from a callback that runs inside the runtime (a progress
handler, or a
+/// session hook of the executions), which may run a query itself: as PySpark
does, the
+/// nested query runs. The thread is then in the runtime's context, which
`block_in_place`
+/// leaves for the nested `block_on`. A `current_thread` runtime cannot block
in place, so the
+/// call keeps failing there.
pub fn block_on<F: std::future::Future>(f: F) -> F::Output {
- get_runtime().block_on(f)
+ match tokio::runtime::Handle::try_current() {
+ Ok(handle) if handle.runtime_flavor() ==
tokio::runtime::RuntimeFlavor::MultiThread => {
+ tokio::task::block_in_place(|| get_runtime().block_on(f))
+ }
+ _ => get_runtime().block_on(f),
+ }
}
#[cfg(test)]
@@ -49,6 +60,20 @@ mod tests {
assert_eq!(result, 42);
}
+ #[test]
+ fn test_block_on_nested_in_block_on() {
+ // A callback of a running execution blocks on a nested query, from
within the runtime.
+ let result = block_on(async { block_on(async { 7 }) + 1 });
+ assert_eq!(result, 8);
+ }
+
+ #[test]
+ fn test_block_on_nested_on_a_worker_thread() {
+ let result =
+ block_on(async { tokio::spawn(async { block_on(async { 5 })
}).await.unwrap() });
+ assert_eq!(result, 5);
+ }
+
#[test]
fn test_block_on_async_task() {
let result = block_on(async {
diff --git a/crates/spark-connect/src/dataframe.rs
b/crates/spark-connect/src/dataframe.rs
index c5a87976..f622d4c5 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -3819,7 +3819,11 @@ mod cache_tests {
.unwrap();
let df = spark.range(3).unwrap();
let cloned = df.clone();
- let runtime = tokio::runtime::Runtime::new().unwrap();
+ // A current-thread runtime: the synchronous API cannot block in place
in it.
+ let runtime = tokio::runtime::Builder::new_current_thread()
+ .enable_all()
+ .build()
+ .unwrap();
// Calling the synchronous API inside a runtime panics during analysis.
// A clone must still attempt analysis afterward, rather than returning
diff --git a/crates/spark-connect/src/session.rs
b/crates/spark-connect/src/session.rs
index 83ce8898..c50906e8 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -2673,10 +2673,14 @@ mod tests {
fn version_cache_recovers_after_lookup_panics() {
let spark = session();
let handle = spark.clone();
- // Like other synchronous RPC helpers, version() cannot run inside
Tokio.
- // That caller's panic must not permanently break this session's cache.
+ // Like other synchronous RPC helpers, version() cannot run inside a
current-thread
+ // Tokio runtime, which cannot block in place. That caller's panic
must not
+ // permanently break this session's cache.
+ let runtime = tokio::runtime::Builder::new_current_thread()
+ .build()
+ .unwrap();
let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
- block_on(async { spark.version() })
+ runtime.block_on(async { spark.version() })
}));
assert!(result.is_err());
assert!(spark.server_version.is_poisoned());
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 16348e4c..d3b79607 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -926,6 +926,76 @@ def
test_foreach_foreach_partition_and_to_json_as_pyspark(session):
assert json.collect() == [Row(value='{"id":1,"name":"a","bin":"eA=="}'),
Row(value='{"id":2}')]
+def test_progress_handler_runs_a_query_of_its_own_as_pyspark(session):
+ # As PySpark's: a handler runs while the execution runs, and a query of
the handler runs
+ # too, the progress of the execution having no effect on it.
+ nested = []
+ started = []
+
+ def handler(*, stages, inflight_tasks, operation_id, done):
+ # The first call, while the result is read, of the progress or of its
end.
+ if not started:
+ started.append(True)
+ nested.append(session.range(4).count())
+
+ session.registerProgressHandler(handler)
+ try:
+ total = (
+ session.range(0, 2000000, 1, 4)
+ .groupBy((F.col("id") % 97).alias("k"))
+ .agg(F.sum("id").alias("s"))
+ .selectExpr("sum(s) as t")
+ .collect()
+ )
+ finally:
+ session.removeProgressHandler(handler)
+ assert nested == [4]
+ assert total == [Row(t=1999999000000)]
+
+
+def test_hook_runs_a_query_of_its_own_as_pyspark(session):
+ # As PySpark's: an on_execute_plan hook runs a query, which runs without
the hook.
+ remote = os.environ["SPARK_REMOTE"]
+ nested = []
+ started = []
+
+ class QueryHook(SparkSession.Hook):
+ def on_execute_plan(self, request):
+ if not started:
+ started.append(True)
+ nested.append(session.range(2).count())
+ return request
+
+ builder = SparkSession.builder.remote(remote)
+ builder._registerHook(lambda _: QueryHook())
+ spark = builder.create()
+ try:
+ assert spark.range(3).count() == 3
+ finally:
+ spark.stop()
+ assert nested == [2]
+
+
+def
test_hook_appending_a_non_reattachable_option_keeps_the_execution_reattachable(session):
+ # As PySpark's: the iterator appends its own reattachable option after the
hooks ran, so
+ # an option of a hook that says false does not make the execution
non-reattachable.
+ remote = os.environ["SPARK_REMOTE"]
+
+ class NotReattachableHook(SparkSession.Hook):
+ def on_execute_plan(self, request):
+ request.request_options.add().reattach_options.reattachable = False
+ return request
+
+ builder = SparkSession.builder.remote(remote)
+ builder._registerHook(lambda _: NotReattachableHook())
+ spark = builder.create()
+ try:
+ assert spark.range(2).count() == 2
+ assert spark.sql("select 1 as x").collect() == [Row(x=1)]
+ finally:
+ spark.stop()
+
+
def test_clone_session_copies_the_state_of_the_session_as_pyspark(session):
# As PySpark's: the clone has the configurations and the temporary views
of the session,
# of an id of its own or of the one given, and changes to either do not
show in the other.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]