This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 06e9e8f [SPARK-60909][CONNECT][FOLLOWUP] Keep spark.sql() lazy within
pipeline flow functions, and release the GIL while it runs a statement
06e9e8f is described below
commit 06e9e8f7374ebdfd3f865cf5917d6cf4bb62ee51
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 12:10:09 2026 +0900
[SPARK-60909][CONNECT][FOLLOWUP] Keep spark.sql() lazy within pipeline flow
functions, and release the GIL while it runs a statement
### What changes were proposed in this pull request?
A follow-up of #152
([SPARK-60909](https://issues.apache.org/jira/browse/SPARK-60909)), which made
`spark.sql()` run its statement at once. A review of the last commits found two
problems with it:
- **Pipeline flow functions:** within a pipeline flow function,
`spark.sql()` now builds its relation without running the statement, as before
#152. The server then analyzes the relation in the pipeline once the
`DefineFlow` defines the flow. PySpark gets the same result by running
`flow.func()` within a `PipelineAnalysisContext`, in which its server returns
the relation of `spark.sql()` without running the statement.
- `SparkConnectGraphElementRegistry.register_flow` enters and leaves a
flow function around `flow.func()`, through `pipeline_enter_flow_function` and
`pipeline_exit_flow_function`.
- The core has a hidden `SparkSession::sql_without_running` for this.
- **The GIL:** `spark.sql()` now releases it while the statement runs, as
the other calls that wait for the server do.
- **Docs:** the Rust `SparkSession::sql_with_references` (#168) now says
that the server creates each name as a temporary view while the statement runs
and drops it after, so a name must not be that of a temporary view to keep.
### Why are the changes needed?
- The registry ran `flow.func()` without a context, as its comment relied
on `spark.sql()` being lazy. A materialized view whose flow is
`spark.sql("SELECT * FROM raw")`, with `raw` a view of the same pipeline,
failed to register with `TABLE_OR_VIEW_NOT_FOUND`, and a statement within a
flow function ran for real.
- While a long statement, such as an `INSERT` or a `CACHE TABLE`, ran, no
other Python thread could run, not even one that would interrupt it.
### Does this PR introduce _any_ user-facing change?
Yes:
- `spark.sql()` within a pipeline flow function defines the flow of its
relation, as in PySpark.
- Other Python threads run while `spark.sql()` runs a statement.
Other requests within a flow function, such as one for the schema of its
DataFrame, are still sent without the pipeline's analysis context. Sending it,
as PySpark does as an extension of the user context, needs the user context of
the requests that #140 adds.
### How was this patch tested?
- A new offline test, with the mock server, defines a materialized view
whose flow function calls `spark.sql()`. No `SqlCommand` is sent, the
`DefineFlow` carries the SQL relation, and a statement outside the flow
function runs at once again.
- A new offline test runs `spark.sql()` on another thread against a server
that holds the statement, and this thread runs meanwhile.
- Both tests fail before this change.
- The lib tests and every integration test, e2e included, against the local
server (643 passed), the build of all tests with `wasm-udf`, the offline Python
suites (615 passed, 8 live cases skipped), and the live drop-in and Arrow
suites (57 passed) and live cases (10 passed). Clippy reports no diagnostics in
the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #173 from HyukjinKwon/SPARK-60909-followup.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/lib.rs | 5 ++
crates/pyspark-rs/src/pipelines.rs | 24 ++++++++
crates/pyspark-rs/src/session.rs | 21 +++++--
crates/spark-connect/src/session.rs | 72 +++++++++++++++-------
.../spark_connect_graph_element_registry.py | 16 +++--
python/tests/_mock_connect_server.py | 8 +++
python/tests/test_dropin_offline.py | 47 ++++++++++++++
7 files changed, 163 insertions(+), 30 deletions(-)
diff --git a/crates/pyspark-rs/src/lib.rs b/crates/pyspark-rs/src/lib.rs
index 1579617..7f632c3 100644
--- a/crates/pyspark-rs/src/lib.rs
+++ b/crates/pyspark-rs/src/lib.rs
@@ -138,6 +138,11 @@ fn _pyspark(_py: Python<'_>, m: &Bound<'_, PyModule>) ->
PyResult<()> {
)?)?;
m.add_function(wrap_pyfunction!(pipelines::pipeline_define_output, m)?)?;
m.add_function(wrap_pyfunction!(pipelines::pipeline_define_flow, m)?)?;
+ m.add_function(wrap_pyfunction!(
+ pipelines::pipeline_enter_flow_function,
+ m
+ )?)?;
+ m.add_function(wrap_pyfunction!(pipelines::pipeline_exit_flow_function,
m)?)?;
m.add_function(wrap_pyfunction!(
pipelines::pipeline_define_auto_cdc_flow,
m
diff --git a/crates/pyspark-rs/src/pipelines.rs
b/crates/pyspark-rs/src/pipelines.rs
index c8f5f55..b508743 100644
--- a/crates/pyspark-rs/src/pipelines.rs
+++ b/crates/pyspark-rs/src/pipelines.rs
@@ -32,6 +32,30 @@ fn scl(file: Option<String>, line: Option<i32>) ->
Option<proto::SourceCodeLocat
})
}
+thread_local! {
+ /// How many pipeline flow functions this thread is in, during which
`spark.sql()` builds
+ /// its relation without running it.
+ static FLOW_FUNCTIONS: std::cell::Cell<usize> = const {
std::cell::Cell::new(0) };
+}
+
+/// Whether this thread is in a pipeline flow function.
+pub(crate) fn in_flow_function() -> bool {
+ FLOW_FUNCTIONS.with(|depth| depth.get() > 0)
+}
+
+/// Enter a pipeline flow function, as PySpark does with a
`PipelineAnalysisContext`, within
+/// which its server returns the relation of `spark.sql()` without running the
statement.
+#[pyfunction]
+pub fn pipeline_enter_flow_function() {
+ FLOW_FUNCTIONS.with(|depth| depth.set(depth.get() + 1));
+}
+
+/// Leave a pipeline flow function.
+#[pyfunction]
+pub fn pipeline_exit_flow_function() {
+ FLOW_FUNCTIONS.with(|depth| depth.set(depth.get().saturating_sub(1)));
+}
+
/// `PipelineCommand.CreateDataflowGraph` → the new graph id.
#[pyfunction]
#[pyo3(signature = (session, default_catalog=None, default_database=None,
sql_conf=None))]
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 4b9ce5c..9fb6011 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -710,10 +710,23 @@ impl PySparkSession {
references.push((df.dataframe.clone(), name));
}
}
- let df = slf
- .borrow()
- .session
- .sql_with_references(&query, pos, named, references)
+ let session = slf.borrow().session.clone();
+ // Within a pipeline flow function, the statement is not run here: the
server analyzes
+ // its relation in the pipeline once the flow is defined, as PySpark's
server returns
+ // the relation without running it within a `PipelineAnalysisContext`.
+ if crate::pipelines::in_flow_function() {
+ return Ok(PyDataFrame::new(
+ session.sql_without_running(&query, pos, named, references),
+ ));
+ }
+ // The statement runs to completion at once, so other Python threads
go on meanwhile,
+ // to interrupt it for one.
+ let df = py
+ .detach(|| {
+ session
+ .sql_with_references(&query, pos, named, references)
+ .map_err(Box::new)
+ })
.to_pyerr()?;
Ok(PyDataFrame::new(df))
}
diff --git a/crates/spark-connect/src/session.rs
b/crates/spark-connect/src/session.rs
index 96c3b1f..000e9d2 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -281,6 +281,11 @@ impl SparkSession {
/// `SparkSession.sql(query, args, **kwargs)` with DataFrames in `kwargs`:
each
/// `(DataFrame, name)` is sent along with the query, as a reference that
the server
/// resolves `name` in the query to.
+ ///
+ /// The server creates each name as a temporary view while it runs the
statement, and
+ /// drops it after, also at each later action on a query's DataFrame. A
name must not be
+ /// that of a temporary view to keep, which the statement would replace
and drop;
+ /// PySpark's `sql()` names its DataFrames
`_pyspark_connect_temp_view_<uuid>` for this.
pub fn sql_with_references(
&self,
query: &str,
@@ -288,27 +293,7 @@ impl SparkSession {
named_args: std::collections::HashMap<String,
crate::expression::Expression>,
references: Vec<(DataFrame, String)>,
) -> Result<DataFrame> {
- let sql = Arc::new(LogicalPlan::Sql {
- query: query.to_string(),
- pos_args,
- named_args,
- });
- let plan = if references.is_empty() {
- sql
- } else {
- Arc::new(LogicalPlan::WithRelations {
- root: sql,
- references: references
- .into_iter()
- .map(|(df, alias)| {
- Arc::new(LogicalPlan::SubqueryAlias {
- input: df.plan,
- alias,
- })
- })
- .collect(),
- })
- };
+ let plan = Self::sql_plan(query, pos_args, named_args, references);
// As pyspark, run the statement as a command now. The server executes
a command and
// returns a relation of its result; for a query it returns the
query's relation, which
// then runs at an action.
@@ -341,6 +326,51 @@ impl SparkSession {
))
}
+ /// The relation of a SQL statement with its parameters and references.
+ fn sql_plan(
+ query: &str,
+ pos_args: Vec<crate::expression::Expression>,
+ named_args: std::collections::HashMap<String,
crate::expression::Expression>,
+ references: Vec<(DataFrame, String)>,
+ ) -> Arc<LogicalPlan> {
+ let sql = Arc::new(LogicalPlan::Sql {
+ query: query.to_string(),
+ pos_args,
+ named_args,
+ });
+ if references.is_empty() {
+ return sql;
+ }
+ Arc::new(LogicalPlan::WithRelations {
+ root: sql,
+ references: references
+ .into_iter()
+ .map(|(df, alias)| {
+ Arc::new(LogicalPlan::SubqueryAlias {
+ input: df.plan,
+ alias,
+ })
+ })
+ .collect(),
+ })
+ }
+
+ /// Internal: the DataFrame of a SQL statement that is not run until an
action, for a
+ /// statement within a pipeline flow function, whose relation the server
analyzes in the
+ /// pipeline instead (PySpark's server returns the relation without
running it there).
+ /// Not a stable API.
+ #[doc(hidden)]
+ pub fn sql_without_running(
+ &self,
+ query: &str,
+ pos_args: Vec<crate::expression::Expression>,
+ named_args: std::collections::HashMap<String,
crate::expression::Expression>,
+ references: Vec<(DataFrame, String)>,
+ ) -> DataFrame {
+ let plan = Self::sql_plan(query, pos_args, named_args, references);
+ DataFrame::new(self.clone(), Arc::unwrap_or_clone(plan))
+ }
+
/// Create a DataFrame from a collection of Rows and a schema.
///
/// Mirrors `pyspark.sql.SparkSession.createDataFrame(rows, schema)`.
diff --git a/python/pyspark/pipelines/spark_connect_graph_element_registry.py
b/python/pyspark/pipelines/spark_connect_graph_element_registry.py
index 25c2315..45dfbc9 100644
--- a/python/pyspark/pipelines/spark_connect_graph_element_registry.py
+++ b/python/pyspark/pipelines/spark_connect_graph_element_registry.py
@@ -41,6 +41,8 @@ from pyspark.sql.types import StructType
from pyspark._pyspark import (
pipeline_define_output as _define_output,
pipeline_define_flow as _define_flow,
+ pipeline_enter_flow_function as _enter_flow_function,
+ pipeline_exit_flow_function as _exit_flow_function,
pipeline_define_auto_cdc_flow as _define_auto_cdc_flow,
pipeline_define_sql_graph_elements as _define_sql_graph_elements,
)
@@ -148,11 +150,15 @@ class
SparkConnectGraphElementRegistry(GraphElementRegistry):
)
def register_flow(self, flow: Flow) -> None:
- # flow.func() builds the query DataFrame lazily (no server
round-trip), so unlike
- # the reference client there is no client-side analysis to scope with a
- # PipelineAnalysisContext; the DefineFlow proto carries the graph id +
flow name,
- # which is what the server uses to analyze the relation.
- df = flow.func()
+ # PySpark runs flow.func() within a PipelineAnalysisContext, in which
its server
+ # returns the relation of spark.sql() without running the statement.
Here spark.sql()
+ # within the flow function builds its relation without running it, and
the server
+ # analyzes the relation in the pipeline once the DefineFlow defines
the flow.
+ _enter_flow_function()
+ try:
+ df = flow.func()
+ finally:
+ _exit_flow_function()
scl_file, scl_line = _scl(flow.source_code_location)
_define_flow(
self._spark,
diff --git a/python/tests/_mock_connect_server.py
b/python/tests/_mock_connect_server.py
index b05469e..b98ba0e 100644
--- a/python/tests/_mock_connect_server.py
+++ b/python/tests/_mock_connect_server.py
@@ -101,6 +101,8 @@ def serve(connection):
failure = None
block_response = False
response_gate = Event()
+ # With `block_execute` set, ExecutePlan waits for the gate too, as
AnalyzePlan does.
+ block_execute = False
# Each AnalyzePlanRequest, recorded.
analyze_requests = []
@@ -113,6 +115,7 @@ def serve(connection):
self.failure = None
self.block_response = False
self.response_gate.clear()
+ self.block_execute = False
self.analyze_requests = []
self.execute_requests = []
self.execute_payload = b""
@@ -164,6 +167,11 @@ def serve(connection):
def execute(self, request, context):
self.execute_calls += 1
self.execute_requests.append(request)
+ if self.block_execute and not self.response_gate.wait(5):
+ context.abort(
+ grpc.StatusCode.DEADLINE_EXCEEDED,
+ "Python thread could not run while the statement ran",
+ )
if self.execute_failure:
context.abort(grpc.StatusCode.INVALID_ARGUMENT,
self.execute_failure)
if self.execute_error:
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index c0ec733..d87c9ad 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -4725,6 +4725,53 @@ def _unparsed_identifiers(expressions):
]
+def
test_connect_registry_defines_a_flow_of_spark_sql_without_running_it(mock_connect):
+ from pyspark import pipelines as dp
+ from pyspark.pipelines.graph_element_registry import
graph_element_registration_context
+ from pyspark.pipelines.spark_connect_graph_element_registry import (
+ SparkConnectGraphElementRegistry,
+ )
+
+ spark, service = mock_connect
+ with
graph_element_registration_context(SparkConnectGraphElementRegistry(spark,
"g")):
+ @dp.materialized_view
+ def mv():
+ return spark.sql("SELECT * FROM raw")
+
+ # As PySpark, whose server returns the relation of spark.sql() within a
flow function
+ # without running the statement: no SqlCommand (Command.sql_command=5) is
sent, and the
+ # flow's relation is the query (DefineFlow.relation_flow_details=7,
relation=1;
+ # Relation.sql=10, SQL.query=1).
+ commands = [parse_message(parse_message(parse_message(r)[3][0])[2][0])
+ for r in service.execute_requests]
+ assert not any(5 in command for command in commands)
+ flow = _sent_define_flow(service.execute_requests[-1])
+ relation = parse_message(parse_message(flow[7][0])[1][0])
+ assert parse_message(relation[10][0])[1] == [b"SELECT * FROM raw"]
+ # Outside of a flow function, the statement runs at once again.
+ spark.sql("SET a=b")
+ assert 5 in
parse_message(parse_message(parse_message(service.execute_requests[-1])[3][0])[2][0])
+
+
+def test_sql_runs_a_statement_without_holding_the_gil(mock_connect):
+ spark, service = mock_connect
+ service.block_execute = True
+ with ThreadPoolExecutor(max_workers=1) as pool:
+ statement = pool.submit(lambda: spark.sql("SET a=b"))
+ try:
+ for _ in range(500):
+ if service.execute_calls:
+ break
+ time.sleep(0.01)
+ else:
+ raise AssertionError("The statement never reached the server")
+ # This thread runs while the statement waits for the server.
+ assert not statement.done()
+ finally:
+ service.release_response()
+ statement.result(timeout=10)
+
+
def test_connect_registry_sends_auto_cdc_define_flow(mock_connect):
from pyspark import pipelines as dp
from pyspark.pipelines.graph_element_registry import
graph_element_registration_context
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]