This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new f543b9f  [SPARK-60909][CONNECT] Run statements at SparkSession.sql() 
as PySpark does
f543b9f is described below

commit f543b9fbdf320f7636b103a17b3b3c27b354bce0
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Thu Nov 5 19:13:46 2026 +0900

    [SPARK-60909][CONNECT] Run statements at SparkSession.sql() as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-60909](https://issues.apache.org/jira/browse/SPARK-60909): 
`SparkSession.sql()` now sends the statement at once, as PySpark Connect 4.3.0 
does, in both the Rust API and the Python client.
    
    - `sql()` sends the statement as a `SqlCommand`, and builds the DataFrame 
from the relation in the server's `SqlCommandResult`, as PySpark's 
`CachedRelation` does. That relation is sent back as is, with the DataFrame's 
own plan ID.
    - For a command, the relation is a local relation of the command's result, 
so the DataFrame is local. For a query, it is the query's relation, which runs 
at an action as before.
    - If the server returns no result relation, the DataFrame keeps the 
statement to run at an action, as in PySpark.
    
    ### Why are the changes needed?
    
    The client built a relation to run at the first action instead, so a 
command took effect only at an action, and ran again at every action:
    
    - After `spark.sql("CREATE OR REPLACE TEMP VIEW v AS SELECT 1 AS x")`, 
`spark.table("v")` failed with `TABLE_OR_VIEW_NOT_FOUND`, and after 
`spark.sql("SET spark.sql.shuffle.partitions=7")` the setting was unchanged.
    - `df = spark.sql("INSERT INTO t VALUES (1)")` inserted nothing, then a row 
at each `df.collect()`, and `df.count()` failed with `INTERNAL_ERROR`.
    - A statement that does not parse or resolve failed only at the first 
action, and `spark.sql("SHOW DATABASES").isLocal()` was false.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, these now match PySpark Connect 4.3.0: a command takes effect at 
`sql()` and runs once, an error in the statement is raised at `sql()`, and a 
command's DataFrame is local. A query's DataFrame still runs at an action.
    
    ### How was this patch tested?
    
    - Against a local Spark 4.3.0 server, probes of a temp view, `SET`, an 
`INSERT` followed by two actions, `args` given as a dict and as a list, and 
`isLocal()` of `SHOW DATABASES` give the same results as official PySpark 4.3.0.
    - New e2e tests, in Rust and Python, check the temp view, `SET` and 
`isLocal()` without an action, and that a syntax error is raised at `sql()`. 
The Python ones also pass with official PySpark 4.3.0.
    - A new unit test checks that a returned relation is sent back with the 
DataFrame's own plan ID.
    - The core lib tests and every integration test, e2e included, against the 
local server (630 passed), the build of all tests with `wasm-udf`, the offline 
Python suites (420 passed, 8 live cases skipped), and the live drop-in and 
Arrow suites (23 passed) and live cases (9 passed). Clippy reports no new 
diagnostics in the changed lines.
    
    The probes also showed differences that this PR does not change, now filed: 
server errors lack their exception class and details, so a syntax error raises 
`AnalysisException` rather than `ParseException` 
([SPARK-60911](https://issues.apache.org/jira/browse/SPARK-60911)); `Row` 
prints as `[1]` rather than `Row(x=1)` 
([SPARK-60912](https://issues.apache.org/jira/browse/SPARK-60912)); and the 
keyword arguments of `sql()` do not format the query 
([SPARK-60913](https://issues.apache.org/ji [...]
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #152 from HyukjinKwon/SPARK-60909-eager-sql.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/spark-connect/src/dataframe.rs             | 55 ++++++++++++++++++++---
 crates/spark-connect/src/plan.rs                  | 10 +++++
 crates/spark-connect/src/session.rs               | 40 +++++++++++++++--
 crates/spark-connect/tests/e2e_session_catalog.rs | 32 +++++++++++++
 docs/architecture.md                              |  5 +++
 python/tests/test_dropin_e2e.py                   | 23 +++++++++-
 6 files changed, 155 insertions(+), 10 deletions(-)

diff --git a/crates/spark-connect/src/dataframe.rs 
b/crates/spark-connect/src/dataframe.rs
index c72d870..a83b0e6 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -1685,12 +1685,22 @@ impl DataFrame {
     ///
     /// Mirrors the server's `Dataset.isLocal`: only a local relation itself 
is local. The
     /// server resolves `toDF`, which Arrow input with inferred or listed 
column names
-    /// applies, into a `Project`, which is not.
+    /// applies, into a `Project`, which is not. The result of a command run by
+    /// `SparkSession::sql` is the local relation the server returned.
     pub fn is_local(&self) -> bool {
-        matches!(
-            self.plan.as_ref(),
-            LogicalPlan::LocalRelation { .. } | 
LogicalPlan::ChunkedCachedLocalRelation { .. }
-        )
+        use proto::relation::RelType;
+        match self.plan.as_ref() {
+            LogicalPlan::LocalRelation { .. } | 
LogicalPlan::ChunkedCachedLocalRelation { .. } => {
+                true
+            }
+            LogicalPlan::CachedRelation { relation } => matches!(
+                relation.rel_type,
+                Some(RelType::LocalRelation(_))
+                    | Some(RelType::CachedLocalRelation(_))
+                    | Some(RelType::ChunkedCachedLocalRelation(_))
+            ),
+            _ => false,
+        }
     }
 
     /// Check if this DataFrame is streaming.
@@ -3126,6 +3136,41 @@ mod plan_construction_tests {
         assert!(!spark.range(1).unwrap().is_local());
     }
 
+    #[test]
+    fn a_relation_the_server_returned_is_sent_back_with_its_own_plan_id() {
+        use proto::relation::RelType;
+        // A session of its own, whose plan IDs start at 1.
+        let spark = session().new_session();
+        let returned = |rel_type: RelType| {
+            let relation = proto::Relation {
+                common: Some(proto::RelationCommon {
+                    plan_id: Some(42),
+                    ..Default::default()
+                }),
+                rel_type: Some(rel_type),
+            };
+            DataFrame::new(
+                spark.clone(),
+                LogicalPlan::CachedRelation {
+                    relation: Box::new(relation),
+                },
+            )
+        };
+        // The result of a command is the local relation the server returned.
+        
assert!(returned(RelType::LocalRelation(proto::LocalRelation::default())).is_local());
+        let query = returned(RelType::Sql(proto::Sql {
+            query: "SELECT 1".to_string(),
+            ..Default::default()
+        }));
+        assert!(!query.is_local());
+        let relation = build_input_relation(&query.plan, &spark);
+        match relation.rel_type {
+            Some(RelType::Sql(sql)) => assert_eq!(sql.query, "SELECT 1"),
+            other => panic!("expected the returned SQL relation, got 
{other:?}"),
+        }
+        assert_eq!(relation.common.and_then(|common| common.plan_id), Some(1));
+    }
+
     #[test]
     fn with_watermark_plan() {
         let spark = session();
diff --git a/crates/spark-connect/src/plan.rs b/crates/spark-connect/src/plan.rs
index 49cc45e..b8f0a59 100644
--- a/crates/spark-connect/src/plan.rs
+++ b/crates/spark-connect/src/plan.rs
@@ -235,6 +235,9 @@ pub enum LogicalPlan {
     },
     /// CachedRemoteRelation: a reference to a cached remote relation.
     CachedRemoteRelation { relation_id: String },
+    /// A relation the server returned, such as the result of 
`SparkSession::sql`, sent back
+    /// as is (pyspark's `CachedRelation`).
+    CachedRelation { relation: Box<proto::Relation> },
     /// Read: `spark.read.format(...).load(...)` or `spark.read.table(...)`.
     Read {
         read_type: crate::readwriter::ReadType,
@@ -1054,6 +1057,13 @@ impl LogicalPlan {
                 relation.rel_type = 
Some(proto::relation::RelType::CachedRemoteRelation(cached));
             }
 
+            LogicalPlan::CachedRelation { relation: returned } => {
+                // Keep the relation as the server returned it; only its plan 
ID is this node's.
+                let common = relation.common.take();
+                relation = (**returned).clone();
+                relation.common = common;
+            }
+
             LogicalPlan::Read {
                 read_type,
                 is_streaming,
diff --git a/crates/spark-connect/src/session.rs 
b/crates/spark-connect/src/session.rs
index 5927b34..eeff006 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -256,9 +256,12 @@ impl SparkSession {
         Ok(DataFrame::new(self.clone(), plan))
     }
 
-    /// Execute a SQL query and return a DataFrame.
+    /// Execute a SQL statement and return a DataFrame.
     ///
-    /// Mirrors `pyspark.sql.SparkSession.sql(sqlQuery)`.
+    /// Mirrors `pyspark.sql.SparkSession.sql(sqlQuery)`: the statement is 
sent at once, so a
+    /// command such as `CREATE VIEW` or `SET` takes effect here and a 
statement that does not
+    /// parse or resolve fails here, while a query still runs only when an 
action is taken on
+    /// the DataFrame.
     pub fn sql(&self, query: &str) -> Result<DataFrame> {
         self.sql_with_args(query, Vec::new(), std::collections::HashMap::new())
     }
@@ -271,12 +274,41 @@ impl SparkSession {
         pos_args: Vec<crate::expression::Expression>,
         named_args: std::collections::HashMap<String, 
crate::expression::Expression>,
     ) -> Result<DataFrame> {
-        let plan = LogicalPlan::Sql {
+        let plan = Arc::new(LogicalPlan::Sql {
             query: query.to_string(),
             pos_args,
             named_args,
+        });
+        // As pyspark, run the statement as a command now. The server executes 
a command and
+        // returns a relation of its result; for a query it returns the 
query's relation, which
+        // then runs at an action.
+        let command = proto::SqlCommand {
+            input: Some(crate::dataframe::build_input_relation(&plan, self)),
+            ..Default::default()
         };
-        Ok(DataFrame::new(self.clone(), plan))
+        let responses = crate::dataframe::execute_command_collect(
+            self,
+            proto::command::CommandType::SqlCommand(command),
+        )?;
+        let returned = responses
+            .into_iter()
+            .find_map(|response| match response.response_type {
+                
Some(proto::execute_plan_response::ResponseType::SqlCommandResult(result)) => {
+                    result.relation
+                }
+                _ => None,
+            });
+        Ok(DataFrame::new(
+            self.clone(),
+            match returned {
+                Some(relation) => LogicalPlan::CachedRelation {
+                    relation: Box::new(relation),
+                },
+                // A server that returns no result relation runs the query 
lazily, as pyspark
+                // then does.
+                None => Arc::unwrap_or_clone(plan),
+            },
+        ))
     }
 
     /// Create a DataFrame from a collection of Rows and a schema.
diff --git a/crates/spark-connect/tests/e2e_session_catalog.rs 
b/crates/spark-connect/tests/e2e_session_catalog.rs
index 1f52f20..cba7b22 100644
--- a/crates/spark-connect/tests/e2e_session_catalog.rs
+++ b/crates/spark-connect/tests/e2e_session_catalog.rs
@@ -225,3 +225,35 @@ fn server_errors_carry_their_condition_and_details() {
     assert_eq!(contexts.len(), 1);
     assert_eq!(contexts[0].fragment(), "1/0");
 }
+
+/// As pyspark, `sql()` sends the statement at once: a command takes effect 
without an
+/// action, its result is the local relation the server returned, and a 
statement that
+/// does not parse fails at `sql()`.
+#[test]
+fn sql_runs_statements_eagerly() {
+    if !should_run() {
+        return;
+    }
+    // A session of its own, so that the view and the setting stay local to 
this test.
+    let spark = session().new_session();
+    spark
+        .sql("CREATE OR REPLACE TEMP VIEW eager_sql_v AS SELECT 1 AS x")
+        .expect("create view");
+    let rows = spark.table("eager_sql_v").unwrap().collect().unwrap();
+    assert_eq!(rows.len(), 1);
+    spark
+        .sql("SET spark.sql.shuffle.partitions=7")
+        .expect("set");
+    assert_eq!(
+        spark.conf().get("spark.sql.shuffle.partitions").unwrap(),
+        Some("7".to_string())
+    );
+    let databases = spark.sql("SHOW DATABASES").expect("show databases");
+    assert!(databases.is_local());
+    assert!(!databases.collect().unwrap().is_empty());
+    let query = spark.sql("SELECT 1 AS x").expect("query");
+    assert!(!query.is_local());
+    assert_eq!(query.collect().unwrap().len(), 1);
+    assert!(spark.sql("SELEC 1").is_err());
+    let _ = spark.stop();
+}
diff --git a/docs/architecture.md b/docs/architecture.md
index 8405256..5cce878 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -110,6 +110,11 @@ last response it saw, so long results survive transient 
disconnects. Like the
 reference client, it releases each response it has read, and the whole 
execution
 once the result is complete, so the server does not keep them.
 
+`spark.sql()` is the exception, as in the reference client: it sends the
+statement at once. A command such as `CREATE VIEW` or `SET` takes effect there,
+and its DataFrame holds the result the server returned; a statement that does 
not
+parse or resolve fails there. A query's DataFrame still runs at an action.
+
 Each plan node gets its plan ID once, the first time it is sent, and keeps it, 
as
 in the reference client. Re-executing a DataFrame therefore sends the same
 relation, and a DataFrame built on another keeps that input's plan IDs, so the
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 6137e18..594dc12 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -26,7 +26,7 @@ from datetime import date, datetime
 
 import pyarrow as pa
 import pytest
-from pyspark.errors import PySparkValueError
+from pyspark.errors import AnalysisException, PySparkValueError
 from pyspark.sql import Row, SparkSession
 from pyspark.sql import functions as F
 from pyspark.sql.types import (
@@ -340,3 +340,24 @@ def test_collected_rows_behave_as_pyspark_rows(session):
     duplicate = session.range(1).selectExpr("1 AS a", "2 AS a").first()
     assert repr(duplicate) == "Row(a=1, a=2)" and duplicate.asDict() == {"a": 
2}
     assert repr(pickle.loads(pickle.dumps(duplicate))) == "Row(a=1, a=2)"
+
+
+def test_sql_runs_statements_eagerly(session):
+    # sql() sends the statement at once: a command takes effect without an 
action, and the
+    # DataFrame of its result is the local relation the server returned.
+    session.sql("CREATE OR REPLACE TEMP VIEW eager_sql_v AS SELECT 1 AS x")
+    assert [tuple(r) for r in session.table("eager_sql_v").collect()] == [(1,)]
+    session.sql("SET spark.connect.rust.test.eager=on")
+    assert session.conf.get("spark.connect.rust.test.eager") == "on"
+    databases = session.sql("SHOW DATABASES")
+    assert databases.isLocal()
+    assert "default" in [r[0] for r in databases.collect()]
+    # A query still runs at the action.
+    query = session.sql("SELECT 1 AS x")
+    assert not query.isLocal()
+    assert [tuple(r) for r in query.collect()] == [(1,)]
+
+
+def test_sql_fails_at_a_statement_that_does_not_parse(session):
+    with pytest.raises(AnalysisException, match="PARSE_SYNTAX_ERROR"):
+        session.sql("SELEC 1")


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to