This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 2b89466f [SPARK-61153][CONNECT][PYTHON] Run the queries of progress 
handlers and hooks, and keep an execution reattachable when a hook adds 
reattach options, as PySpark does
2b89466f is described below

commit 2b89466f97855cae890a9fe96986f2082a60430f
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 12:47:02 2026 +0900

    [SPARK-61153][CONNECT][PYTHON] Run the queries of progress handlers and 
hooks, and keep an execution reattachable when a hook adds reattach options, as 
PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61153](https://issues.apache.org/jira/browse/SPARK-61153): a query 
run inside a progress handler or a `SparkSession.Hook` runs, and a hook that 
adds `reattachable = False` keeps the execution reattachable, as PySpark does.
    
    - `spark_connect_core::runtime::block_on` blocks in place when it is called 
from a thread that is already in the shared multi-thread runtime's context, 
which is the case for a callback that runs inside an execution. The nested 
query then runs, as it does in PySpark. On a `current_thread` runtime the call 
still fails as before, since such a runtime cannot block in place. No lock is 
held across the callback: the progress handlers and the hook are cloned out of 
their locks before they ar [...]
    - `ExecutePlanResponseReattachableIterator::new` always appends its 
`reattachable = True` option, after the request options that the hooks added, 
as PySpark's `ExecutePlanResponseReattachableIterator` does. Before, it 
appended the option only when the request had none, so a hook's `reattachable = 
False` made the server treat the execution as not reattachable, and the rows 
were lost with `INVALID_HANDLE.OPERATION_NOT_FOUND`.
    - Two unit tests that relied on the nested-runtime panic now use a 
`current_thread` runtime, which keeps the panic they check: the schema cache of 
a dataframe and the version cache of a session.
    
    ### Why are the changes needed?
    
    Both defects run Python callbacks while an execution runs:
    
    ```python
    s.registerProgressHandler(lambda **kw: s.range(4).count() if not kw["done"] 
else None)
    s.range(2000000).selectExpr("sum(id) as t").collect()
    # PanicException: Cannot start a runtime from within a runtime
    
    class H(SparkSession.Hook):
        def on_execute_plan(self, request):
            other.range(2).count()   # the same panic
            return request
    
    b._registerHook(lambda s: H())
    
    b._registerHook(lambda s: AppendNotReattachable())   # appends 
request_options reattach_options.reattachable = False
    s.range(2).count()
    # INVALID_HANDLE.OPERATION_NOT_FOUND after the rows are read
    ```
    
    Official PySpark 4.3.0 gives `nested count in handler -> 4`, `outer -> 
[Row(t=1999999000000)]`, `outer hook query -> 3`, and `(2, [Row(x=1)])` for the 
last case.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. A query run in a progress handler or a session hook returns its result 
instead of panicking, and a hook that appends a non-reattachable option no 
longer fails the execution.
    
    ### How was this patch tested?
    
    - Rust `test_block_on_nested_in_block_on` and 
`test_block_on_nested_on_a_worker_thread` in `runtime.rs`: a `block_on` from 
within `block_on`, and from a worker thread.
    - Rust `test_reattach_options_set_after_a_hook_added_a_false_one` in 
`reattach.rs`: the iterator's options are `[false, true]` after a hook added 
`false`.
    - Live `test_progress_handler_runs_a_query_of_its_own_as_pyspark`: a 
handler runs a count while the execution runs. The same results with official 
PySpark 4.3.0.
    - Live `test_hook_runs_a_query_of_its_own_as_pyspark`: a hook runs a query. 
The same results with official PySpark 4.3.0.
    - Live 
`test_hook_appending_a_non_reattachable_option_keeps_the_execution_reattachable`:
 a count and a `sql` collect with a hook that appends `reattachable = False`. 
The same results with official PySpark 4.3.0.
    - The tests of both crates, e2e included, against a local Spark 4.3.0 
server (878 passed), the build of all tests, also with `wasm-udf`, the offline 
Python suites (2201 passed, 8 skipped), and the live drop-in, Arrow and ML 
suites (187 passed) and live cases (10 passed). rustfmt passes, and clippy 
reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #342 from HyukjinKwon/SPARK-61153-nested-queries-hooks-reattach.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/spark-connect-core/src/reattach.rs | 58 ++++++++++++++++++-------
 crates/spark-connect-core/src/runtime.rs  | 27 +++++++++++-
 crates/spark-connect/src/dataframe.rs     |  6 ++-
 crates/spark-connect/src/session.rs       | 10 +++--
 python/tests/test_dropin_e2e.py           | 70 +++++++++++++++++++++++++++++++
 5 files changed, 152 insertions(+), 19 deletions(-)

diff --git a/crates/spark-connect-core/src/reattach.rs 
b/crates/spark-connect-core/src/reattach.rs
index d5c20281..e21feb38 100644
--- a/crates/spark-connect-core/src/reattach.rs
+++ b/crates/spark-connect-core/src/reattach.rs
@@ -43,20 +43,18 @@ impl ExecutePlanResponseReattachableIterator {
 
         request.operation_id = Some(operation_id.clone());
 
-        // Ensure reattachable options are set
-        if !request.request_options.iter().any(|opt| {
-            matches!(
-                &opt.request_option,
-                
Some(spark_connect_proto::execute_plan_request::request_option::RequestOption::ReattachOptions(_))
-            )
-        }) {
-            let mut req_opt = 
spark_connect_proto::execute_plan_request::RequestOption::default();
-            req_opt.request_option =
-                
Some(spark_connect_proto::execute_plan_request::request_option::RequestOption::ReattachOptions(
-                    spark_connect_proto::ReattachOptions { reattachable: true 
},
-                ));
-            request.request_options.push(req_opt);
-        }
+        // Always append the reattachable option, as PySpark does: a hook may 
have added one
+        // with `reattachable = False`, which the server takes as not 
reattachable only when no
+        // option says true.
+        request
+            .request_options
+            .push(spark_connect_proto::execute_plan_request::RequestOption {
+                request_option: Some(
+                    
spark_connect_proto::execute_plan_request::request_option::RequestOption::ReattachOptions(
+                        spark_connect_proto::ReattachOptions { reattachable: 
true },
+                    ),
+                ),
+            });
 
         Self {
             operation_id,
@@ -245,6 +243,38 @@ mod tests {
         }));
     }
 
+    #[tokio::test]
+    async fn test_reattach_options_set_after_a_hook_added_a_false_one() {
+        // A session hook appends `reattachable = False`; the execution still 
ends up with an
+        // option that says true, as PySpark's iterator appends its own after 
the hooks ran.
+        let request = ExecutePlanRequest {
+            session_id: "session-123".to_string(),
+            request_options: 
vec![spark_connect_proto::execute_plan_request::RequestOption {
+                request_option: Some(
+                    
spark_connect_proto::execute_plan_request::request_option::RequestOption::ReattachOptions(
+                        spark_connect_proto::ReattachOptions {
+                            reattachable: false,
+                        },
+                    ),
+                ),
+            }],
+            ..Default::default()
+        };
+
+        let iter = ExecutePlanResponseReattachableIterator::new(request);
+
+        let reattachable: Vec<bool> = iter
+            .request()
+            .request_options
+            .iter()
+            .filter_map(|opt| match &opt.request_option {
+                
Some(spark_connect_proto::execute_plan_request::request_option::RequestOption::ReattachOptions(ro))
 => Some(ro.reattachable),
+                _ => None,
+            })
+            .collect();
+        assert_eq!(reattachable, vec![false, true]);
+    }
+
     #[tokio::test]
     async fn test_create_reattach_request() {
         let mut request = ExecutePlanRequest::default();
diff --git a/crates/spark-connect-core/src/runtime.rs 
b/crates/spark-connect-core/src/runtime.rs
index 070c6f5c..b694f281 100644
--- a/crates/spark-connect-core/src/runtime.rs
+++ b/crates/spark-connect-core/src/runtime.rs
@@ -27,8 +27,19 @@ pub fn get_runtime() -> &'static Runtime {
 /// Allows synchronous code (e.g., from PyO3) to execute an async function
 /// and wait for its result. This should be called with the GIL released
 /// in a Python context.
+///
+/// Also called from a callback that runs inside the runtime (a progress 
handler, or a
+/// session hook of the executions), which may run a query itself: as PySpark 
does, the
+/// nested query runs. The thread is then in the runtime's context, which 
`block_in_place`
+/// leaves for the nested `block_on`. A `current_thread` runtime cannot block 
in place, so the
+/// call keeps failing there.
 pub fn block_on<F: std::future::Future>(f: F) -> F::Output {
-    get_runtime().block_on(f)
+    match tokio::runtime::Handle::try_current() {
+        Ok(handle) if handle.runtime_flavor() == 
tokio::runtime::RuntimeFlavor::MultiThread => {
+            tokio::task::block_in_place(|| get_runtime().block_on(f))
+        }
+        _ => get_runtime().block_on(f),
+    }
 }
 
 #[cfg(test)]
@@ -49,6 +60,20 @@ mod tests {
         assert_eq!(result, 42);
     }
 
+    #[test]
+    fn test_block_on_nested_in_block_on() {
+        // A callback of a running execution blocks on a nested query, from 
within the runtime.
+        let result = block_on(async { block_on(async { 7 }) + 1 });
+        assert_eq!(result, 8);
+    }
+
+    #[test]
+    fn test_block_on_nested_on_a_worker_thread() {
+        let result =
+            block_on(async { tokio::spawn(async { block_on(async { 5 }) 
}).await.unwrap() });
+        assert_eq!(result, 5);
+    }
+
     #[test]
     fn test_block_on_async_task() {
         let result = block_on(async {
diff --git a/crates/spark-connect/src/dataframe.rs 
b/crates/spark-connect/src/dataframe.rs
index c5a87976..f622d4c5 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -3819,7 +3819,11 @@ mod cache_tests {
             .unwrap();
         let df = spark.range(3).unwrap();
         let cloned = df.clone();
-        let runtime = tokio::runtime::Runtime::new().unwrap();
+        // A current-thread runtime: the synchronous API cannot block in place 
in it.
+        let runtime = tokio::runtime::Builder::new_current_thread()
+            .enable_all()
+            .build()
+            .unwrap();
 
         // Calling the synchronous API inside a runtime panics during analysis.
         // A clone must still attempt analysis afterward, rather than returning
diff --git a/crates/spark-connect/src/session.rs 
b/crates/spark-connect/src/session.rs
index 83ce8898..c50906e8 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -2673,10 +2673,14 @@ mod tests {
     fn version_cache_recovers_after_lookup_panics() {
         let spark = session();
         let handle = spark.clone();
-        // Like other synchronous RPC helpers, version() cannot run inside 
Tokio.
-        // That caller's panic must not permanently break this session's cache.
+        // Like other synchronous RPC helpers, version() cannot run inside a 
current-thread
+        // Tokio runtime, which cannot block in place. That caller's panic 
must not
+        // permanently break this session's cache.
+        let runtime = tokio::runtime::Builder::new_current_thread()
+            .build()
+            .unwrap();
         let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
-            block_on(async { spark.version() })
+            runtime.block_on(async { spark.version() })
         }));
         assert!(result.is_err());
         assert!(spark.server_version.is_poisoned());
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 16348e4c..d3b79607 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -926,6 +926,76 @@ def 
test_foreach_foreach_partition_and_to_json_as_pyspark(session):
     assert json.collect() == [Row(value='{"id":1,"name":"a","bin":"eA=="}'), 
Row(value='{"id":2}')]
 
 
+def test_progress_handler_runs_a_query_of_its_own_as_pyspark(session):
+    # As PySpark's: a handler runs while the execution runs, and a query of 
the handler runs
+    # too, the progress of the execution having no effect on it.
+    nested = []
+    started = []
+
+    def handler(*, stages, inflight_tasks, operation_id, done):
+        # The first call, while the result is read, of the progress or of its 
end.
+        if not started:
+            started.append(True)
+            nested.append(session.range(4).count())
+
+    session.registerProgressHandler(handler)
+    try:
+        total = (
+            session.range(0, 2000000, 1, 4)
+            .groupBy((F.col("id") % 97).alias("k"))
+            .agg(F.sum("id").alias("s"))
+            .selectExpr("sum(s) as t")
+            .collect()
+        )
+    finally:
+        session.removeProgressHandler(handler)
+    assert nested == [4]
+    assert total == [Row(t=1999999000000)]
+
+
+def test_hook_runs_a_query_of_its_own_as_pyspark(session):
+    # As PySpark's: an on_execute_plan hook runs a query, which runs without 
the hook.
+    remote = os.environ["SPARK_REMOTE"]
+    nested = []
+    started = []
+
+    class QueryHook(SparkSession.Hook):
+        def on_execute_plan(self, request):
+            if not started:
+                started.append(True)
+                nested.append(session.range(2).count())
+            return request
+
+    builder = SparkSession.builder.remote(remote)
+    builder._registerHook(lambda _: QueryHook())
+    spark = builder.create()
+    try:
+        assert spark.range(3).count() == 3
+    finally:
+        spark.stop()
+    assert nested == [2]
+
+
+def 
test_hook_appending_a_non_reattachable_option_keeps_the_execution_reattachable(session):
+    # As PySpark's: the iterator appends its own reattachable option after the 
hooks ran, so
+    # an option of a hook that says false does not make the execution 
non-reattachable.
+    remote = os.environ["SPARK_REMOTE"]
+
+    class NotReattachableHook(SparkSession.Hook):
+        def on_execute_plan(self, request):
+            request.request_options.add().reattach_options.reattachable = False
+            return request
+
+    builder = SparkSession.builder.remote(remote)
+    builder._registerHook(lambda _: NotReattachableHook())
+    spark = builder.create()
+    try:
+        assert spark.range(2).count() == 2
+        assert spark.sql("select 1 as x").collect() == [Row(x=1)]
+    finally:
+        spark.stop()
+
+
 def test_clone_session_copies_the_state_of_the_session_as_pyspark(session):
     # As PySpark's: the clone has the configurations and the temporary views 
of the session,
     # of an id of its own or of the one given, and changes to either do not 
show in the other.


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to