This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 06e9e8f  [SPARK-60909][CONNECT][FOLLOWUP] Keep spark.sql() lazy within 
pipeline flow functions, and release the GIL while it runs a statement
06e9e8f is described below

commit 06e9e8f7374ebdfd3f865cf5917d6cf4bb62ee51
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 12:10:09 2026 +0900

    [SPARK-60909][CONNECT][FOLLOWUP] Keep spark.sql() lazy within pipeline flow 
functions, and release the GIL while it runs a statement
    
    ### What changes were proposed in this pull request?
    
    A follow-up of #152 
([SPARK-60909](https://issues.apache.org/jira/browse/SPARK-60909)), which made 
`spark.sql()` run its statement at once. A review of the last commits found two 
problems with it:
    
    - **Pipeline flow functions:** within a pipeline flow function, 
`spark.sql()` now builds its relation without running the statement, as before 
#152. The server then analyzes the relation in the pipeline once the 
`DefineFlow` defines the flow. PySpark gets the same result by running 
`flow.func()` within a `PipelineAnalysisContext`, in which its server returns 
the relation of `spark.sql()` without running the statement.
      - `SparkConnectGraphElementRegistry.register_flow` enters and leaves a 
flow function around `flow.func()`, through `pipeline_enter_flow_function` and 
`pipeline_exit_flow_function`.
      - The core has a hidden `SparkSession::sql_without_running` for this.
    - **The GIL:** `spark.sql()` now releases it while the statement runs, as 
the other calls that wait for the server do.
    - **Docs:** the Rust `SparkSession::sql_with_references` (#168) now says 
that the server creates each name as a temporary view while the statement runs 
and drops it after, so a name must not be that of a temporary view to keep.
    
    ### Why are the changes needed?
    
    - The registry ran `flow.func()` without a context, as its comment relied 
on `spark.sql()` being lazy. A materialized view whose flow is 
`spark.sql("SELECT * FROM raw")`, with `raw` a view of the same pipeline, 
failed to register with `TABLE_OR_VIEW_NOT_FOUND`, and a statement within a 
flow function ran for real.
    - While a long statement, such as an `INSERT` or a `CACHE TABLE`, ran, no 
other Python thread could run, not even one that would interrupt it.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes:
    - `spark.sql()` within a pipeline flow function defines the flow of its 
relation, as in PySpark.
    - Other Python threads run while `spark.sql()` runs a statement.
    
    Other requests within a flow function, such as one for the schema of its 
DataFrame, are still sent without the pipeline's analysis context. Sending it, 
as PySpark does as an extension of the user context, needs the user context of 
the requests that #140 adds.
    
    ### How was this patch tested?
    
    - A new offline test, with the mock server, defines a materialized view 
whose flow function calls `spark.sql()`. No `SqlCommand` is sent, the 
`DefineFlow` carries the SQL relation, and a statement outside the flow 
function runs at once again.
    - A new offline test runs `spark.sql()` on another thread against a server 
that holds the statement, and this thread runs meanwhile.
    - Both tests fail before this change.
    - The lib tests and every integration test, e2e included, against the local 
server (643 passed), the build of all tests with `wasm-udf`, the offline Python 
suites (615 passed, 8 live cases skipped), and the live drop-in and Arrow 
suites (57 passed) and live cases (10 passed). Clippy reports no diagnostics in 
the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #173 from HyukjinKwon/SPARK-60909-followup.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/lib.rs                       |  5 ++
 crates/pyspark-rs/src/pipelines.rs                 | 24 ++++++++
 crates/pyspark-rs/src/session.rs                   | 21 +++++--
 crates/spark-connect/src/session.rs                | 72 +++++++++++++++-------
 .../spark_connect_graph_element_registry.py        | 16 +++--
 python/tests/_mock_connect_server.py               |  8 +++
 python/tests/test_dropin_offline.py                | 47 ++++++++++++++
 7 files changed, 163 insertions(+), 30 deletions(-)

diff --git a/crates/pyspark-rs/src/lib.rs b/crates/pyspark-rs/src/lib.rs
index 1579617..7f632c3 100644
--- a/crates/pyspark-rs/src/lib.rs
+++ b/crates/pyspark-rs/src/lib.rs
@@ -138,6 +138,11 @@ fn _pyspark(_py: Python<'_>, m: &Bound<'_, PyModule>) -> 
PyResult<()> {
     )?)?;
     m.add_function(wrap_pyfunction!(pipelines::pipeline_define_output, m)?)?;
     m.add_function(wrap_pyfunction!(pipelines::pipeline_define_flow, m)?)?;
+    m.add_function(wrap_pyfunction!(
+        pipelines::pipeline_enter_flow_function,
+        m
+    )?)?;
+    m.add_function(wrap_pyfunction!(pipelines::pipeline_exit_flow_function, 
m)?)?;
     m.add_function(wrap_pyfunction!(
         pipelines::pipeline_define_auto_cdc_flow,
         m
diff --git a/crates/pyspark-rs/src/pipelines.rs 
b/crates/pyspark-rs/src/pipelines.rs
index c8f5f55..b508743 100644
--- a/crates/pyspark-rs/src/pipelines.rs
+++ b/crates/pyspark-rs/src/pipelines.rs
@@ -32,6 +32,30 @@ fn scl(file: Option<String>, line: Option<i32>) -> 
Option<proto::SourceCodeLocat
     })
 }
 
+thread_local! {
+    /// How many pipeline flow functions this thread is in, during which 
`spark.sql()` builds
+    /// its relation without running it.
+    static FLOW_FUNCTIONS: std::cell::Cell<usize> = const { 
std::cell::Cell::new(0) };
+}
+
+/// Whether this thread is in a pipeline flow function.
+pub(crate) fn in_flow_function() -> bool {
+    FLOW_FUNCTIONS.with(|depth| depth.get() > 0)
+}
+
+/// Enter a pipeline flow function, as PySpark does with a 
`PipelineAnalysisContext`, within
+/// which its server returns the relation of `spark.sql()` without running the 
statement.
+#[pyfunction]
+pub fn pipeline_enter_flow_function() {
+    FLOW_FUNCTIONS.with(|depth| depth.set(depth.get() + 1));
+}
+
+/// Leave a pipeline flow function.
+#[pyfunction]
+pub fn pipeline_exit_flow_function() {
+    FLOW_FUNCTIONS.with(|depth| depth.set(depth.get().saturating_sub(1)));
+}
+
 /// `PipelineCommand.CreateDataflowGraph` → the new graph id.
 #[pyfunction]
 #[pyo3(signature = (session, default_catalog=None, default_database=None, 
sql_conf=None))]
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 4b9ce5c..9fb6011 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -710,10 +710,23 @@ impl PySparkSession {
                 references.push((df.dataframe.clone(), name));
             }
         }
-        let df = slf
-            .borrow()
-            .session
-            .sql_with_references(&query, pos, named, references)
+        let session = slf.borrow().session.clone();
+        // Within a pipeline flow function, the statement is not run here: the 
server analyzes
+        // its relation in the pipeline once the flow is defined, as PySpark's 
server returns
+        // the relation without running it within a `PipelineAnalysisContext`.
+        if crate::pipelines::in_flow_function() {
+            return Ok(PyDataFrame::new(
+                session.sql_without_running(&query, pos, named, references),
+            ));
+        }
+        // The statement runs to completion at once, so other Python threads 
go on meanwhile,
+        // to interrupt it for one.
+        let df = py
+            .detach(|| {
+                session
+                    .sql_with_references(&query, pos, named, references)
+                    .map_err(Box::new)
+            })
             .to_pyerr()?;
         Ok(PyDataFrame::new(df))
     }
diff --git a/crates/spark-connect/src/session.rs 
b/crates/spark-connect/src/session.rs
index 96c3b1f..000e9d2 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -281,6 +281,11 @@ impl SparkSession {
     /// `SparkSession.sql(query, args, **kwargs)` with DataFrames in `kwargs`: 
each
     /// `(DataFrame, name)` is sent along with the query, as a reference that 
the server
     /// resolves `name` in the query to.
+    ///
+    /// The server creates each name as a temporary view while it runs the 
statement, and
+    /// drops it after, also at each later action on a query's DataFrame. A 
name must not be
+    /// that of a temporary view to keep, which the statement would replace 
and drop;
+    /// PySpark's `sql()` names its DataFrames 
`_pyspark_connect_temp_view_<uuid>` for this.
     pub fn sql_with_references(
         &self,
         query: &str,
@@ -288,27 +293,7 @@ impl SparkSession {
         named_args: std::collections::HashMap<String, 
crate::expression::Expression>,
         references: Vec<(DataFrame, String)>,
     ) -> Result<DataFrame> {
-        let sql = Arc::new(LogicalPlan::Sql {
-            query: query.to_string(),
-            pos_args,
-            named_args,
-        });
-        let plan = if references.is_empty() {
-            sql
-        } else {
-            Arc::new(LogicalPlan::WithRelations {
-                root: sql,
-                references: references
-                    .into_iter()
-                    .map(|(df, alias)| {
-                        Arc::new(LogicalPlan::SubqueryAlias {
-                            input: df.plan,
-                            alias,
-                        })
-                    })
-                    .collect(),
-            })
-        };
+        let plan = Self::sql_plan(query, pos_args, named_args, references);
         // As pyspark, run the statement as a command now. The server executes 
a command and
         // returns a relation of its result; for a query it returns the 
query's relation, which
         // then runs at an action.
@@ -341,6 +326,51 @@ impl SparkSession {
         ))
     }
 
+    /// The relation of a SQL statement with its parameters and references.
+    fn sql_plan(
+        query: &str,
+        pos_args: Vec<crate::expression::Expression>,
+        named_args: std::collections::HashMap<String, 
crate::expression::Expression>,
+        references: Vec<(DataFrame, String)>,
+    ) -> Arc<LogicalPlan> {
+        let sql = Arc::new(LogicalPlan::Sql {
+            query: query.to_string(),
+            pos_args,
+            named_args,
+        });
+        if references.is_empty() {
+            return sql;
+        }
+        Arc::new(LogicalPlan::WithRelations {
+            root: sql,
+            references: references
+                .into_iter()
+                .map(|(df, alias)| {
+                    Arc::new(LogicalPlan::SubqueryAlias {
+                        input: df.plan,
+                        alias,
+                    })
+                })
+                .collect(),
+        })
+    }
+
+    /// Internal: the DataFrame of a SQL statement that is not run until an 
action, for a
+    /// statement within a pipeline flow function, whose relation the server 
analyzes in the
+    /// pipeline instead (PySpark's server returns the relation without 
running it there).
+    /// Not a stable API.
+    #[doc(hidden)]
+    pub fn sql_without_running(
+        &self,
+        query: &str,
+        pos_args: Vec<crate::expression::Expression>,
+        named_args: std::collections::HashMap<String, 
crate::expression::Expression>,
+        references: Vec<(DataFrame, String)>,
+    ) -> DataFrame {
+        let plan = Self::sql_plan(query, pos_args, named_args, references);
+        DataFrame::new(self.clone(), Arc::unwrap_or_clone(plan))
+    }
+
     /// Create a DataFrame from a collection of Rows and a schema.
     ///
     /// Mirrors `pyspark.sql.SparkSession.createDataFrame(rows, schema)`.
diff --git a/python/pyspark/pipelines/spark_connect_graph_element_registry.py 
b/python/pyspark/pipelines/spark_connect_graph_element_registry.py
index 25c2315..45dfbc9 100644
--- a/python/pyspark/pipelines/spark_connect_graph_element_registry.py
+++ b/python/pyspark/pipelines/spark_connect_graph_element_registry.py
@@ -41,6 +41,8 @@ from pyspark.sql.types import StructType
 from pyspark._pyspark import (
     pipeline_define_output as _define_output,
     pipeline_define_flow as _define_flow,
+    pipeline_enter_flow_function as _enter_flow_function,
+    pipeline_exit_flow_function as _exit_flow_function,
     pipeline_define_auto_cdc_flow as _define_auto_cdc_flow,
     pipeline_define_sql_graph_elements as _define_sql_graph_elements,
 )
@@ -148,11 +150,15 @@ class 
SparkConnectGraphElementRegistry(GraphElementRegistry):
             )
 
     def register_flow(self, flow: Flow) -> None:
-        # flow.func() builds the query DataFrame lazily (no server 
round-trip), so unlike
-        # the reference client there is no client-side analysis to scope with a
-        # PipelineAnalysisContext; the DefineFlow proto carries the graph id + 
flow name,
-        # which is what the server uses to analyze the relation.
-        df = flow.func()
+        # PySpark runs flow.func() within a PipelineAnalysisContext, in which 
its server
+        # returns the relation of spark.sql() without running the statement. 
Here spark.sql()
+        # within the flow function builds its relation without running it, and 
the server
+        # analyzes the relation in the pipeline once the DefineFlow defines 
the flow.
+        _enter_flow_function()
+        try:
+            df = flow.func()
+        finally:
+            _exit_flow_function()
         scl_file, scl_line = _scl(flow.source_code_location)
         _define_flow(
             self._spark,
diff --git a/python/tests/_mock_connect_server.py 
b/python/tests/_mock_connect_server.py
index b05469e..b98ba0e 100644
--- a/python/tests/_mock_connect_server.py
+++ b/python/tests/_mock_connect_server.py
@@ -101,6 +101,8 @@ def serve(connection):
         failure = None
         block_response = False
         response_gate = Event()
+        # With `block_execute` set, ExecutePlan waits for the gate too, as 
AnalyzePlan does.
+        block_execute = False
 
         # Each AnalyzePlanRequest, recorded.
         analyze_requests = []
@@ -113,6 +115,7 @@ def serve(connection):
             self.failure = None
             self.block_response = False
             self.response_gate.clear()
+            self.block_execute = False
             self.analyze_requests = []
             self.execute_requests = []
             self.execute_payload = b""
@@ -164,6 +167,11 @@ def serve(connection):
         def execute(self, request, context):
             self.execute_calls += 1
             self.execute_requests.append(request)
+            if self.block_execute and not self.response_gate.wait(5):
+                context.abort(
+                    grpc.StatusCode.DEADLINE_EXCEEDED,
+                    "Python thread could not run while the statement ran",
+                )
             if self.execute_failure:
                 context.abort(grpc.StatusCode.INVALID_ARGUMENT, 
self.execute_failure)
             if self.execute_error:
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index c0ec733..d87c9ad 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -4725,6 +4725,53 @@ def _unparsed_identifiers(expressions):
     ]
 
 
+def 
test_connect_registry_defines_a_flow_of_spark_sql_without_running_it(mock_connect):
+    from pyspark import pipelines as dp
+    from pyspark.pipelines.graph_element_registry import 
graph_element_registration_context
+    from pyspark.pipelines.spark_connect_graph_element_registry import (
+        SparkConnectGraphElementRegistry,
+    )
+
+    spark, service = mock_connect
+    with 
graph_element_registration_context(SparkConnectGraphElementRegistry(spark, 
"g")):
+        @dp.materialized_view
+        def mv():
+            return spark.sql("SELECT * FROM raw")
+
+    # As PySpark, whose server returns the relation of spark.sql() within a 
flow function
+    # without running the statement: no SqlCommand (Command.sql_command=5) is 
sent, and the
+    # flow's relation is the query (DefineFlow.relation_flow_details=7, 
relation=1;
+    # Relation.sql=10, SQL.query=1).
+    commands = [parse_message(parse_message(parse_message(r)[3][0])[2][0])
+                for r in service.execute_requests]
+    assert not any(5 in command for command in commands)
+    flow = _sent_define_flow(service.execute_requests[-1])
+    relation = parse_message(parse_message(flow[7][0])[1][0])
+    assert parse_message(relation[10][0])[1] == [b"SELECT * FROM raw"]
+    # Outside of a flow function, the statement runs at once again.
+    spark.sql("SET a=b")
+    assert 5 in 
parse_message(parse_message(parse_message(service.execute_requests[-1])[3][0])[2][0])
+
+
+def test_sql_runs_a_statement_without_holding_the_gil(mock_connect):
+    spark, service = mock_connect
+    service.block_execute = True
+    with ThreadPoolExecutor(max_workers=1) as pool:
+        statement = pool.submit(lambda: spark.sql("SET a=b"))
+        try:
+            for _ in range(500):
+                if service.execute_calls:
+                    break
+                time.sleep(0.01)
+            else:
+                raise AssertionError("The statement never reached the server")
+            # This thread runs while the statement waits for the server.
+            assert not statement.done()
+        finally:
+            service.release_response()
+        statement.result(timeout=10)
+
+
 def test_connect_registry_sends_auto_cdc_define_flow(mock_connect):
     from pyspark import pipelines as dp
     from pyspark.pipelines.graph_element_registry import 
graph_element_registration_context


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to