This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 014674e  [SPARK-60913][CONNECT] Format spark.sql() queries with 
keyword arguments as PySpark does
014674e is described below

commit 014674e44470001f5c184846964c131d43f6e4fa
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 08:46:32 2026 +0900

    [SPARK-60913][CONNECT] Format spark.sql() queries with keyword arguments as 
PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-60913](https://issues.apache.org/jira/browse/SPARK-60913): the 
keyword arguments of `SparkSession.sql(sqlQuery, args=None, **kwargs)` now 
format the query, as in PySpark Connect.
    
    - The query is formatted as `str.format` does, with the formatter of 
`pyspark.sql.connect.sql_formatter`, a shim that mirrors PySpark's:
      - A DataFrame becomes a generated name, 
`_pyspark_connect_temp_view_<uuid>`, one for each DataFrame.
      - A plain column reference, such as `col("id")` or `df.id`, becomes its 
name. Another expression raises `PySparkValueError`.
      - A string becomes its SQL literal, and another value its string.
    - Each DataFrame is sent with the statement as a reference that the server 
resolves its name to. The statement's relation is then a `WithRelations` of the 
query with a `SubqueryAlias` of each DataFrame, as in PySpark.
    - `args` still binds parameters, and now raises `INVALID_TYPE` when it is 
neither a dict nor a list, as in PySpark.
    - In the Rust core:
      - `LogicalPlan::WithRelations { root, references }` is sent as the 
`with_relations` relation.
      - `SparkSession::sql_with_references` takes the `(DataFrame, name)` pairs 
that the query names, and `sql_with_args` calls it without any.
    - In the Python client, `pyspark._pyspark.column_reference_name` tells the 
formatter whether a Column is a plain column reference, and gives its name. It 
is a function rather than an attribute, as the attributes of a PySpark Column 
are its fields 
([SPARK-60949](https://issues.apache.org/jira/browse/SPARK-60949)).
    
    ### Why are the changes needed?
    
    The keyword arguments were bound as named parameters, and the query was 
sent as is:
    
    - `spark.sql("SELECT {x} AS v", x=1)` failed with `PARSE_SYNTAX_ERROR`, 
rather than returning 1.
    - `spark.sql("SELECT count(*) AS n FROM {df}", df=spark.range(3))` raised 
`TypeError`, rather than returning 3.
    - `spark.sql("SELECT :p AS v", p=5)` returned 5, where PySpark fails with 
`UNBOUND_SQL_PARAMETER`, as `p` is not a parameter.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, `sql()` with keyword arguments now behaves as in PySpark Connect 
4.3.0. A keyword argument is no longer a named parameter; `args` binds those.
    
    ### How was this patch tested?
    
    - Against a local Spark 4.3.0 server, a probe of nineteen cases gives 
output identical to official PySpark 4.3.0, apart from one. The cases cover:
      - values, strings with quotes and backslashes, `col()`, `df.col`, and 
format specs;
      - one DataFrame named twice, and a join of two;
      - `args` together with keyword arguments, and `args` as a tuple;
      - escaped braces with and without keyword arguments, and a command that 
reads a DataFrame;
      - the formatter's own errors.
    
      The one difference is the text of the Column in the error for an 
expression that is not a column reference: `Column<'(id + 1)'>` rather than 
`Column<'+(id, 1)'>`, which 
[SPARK-60948](https://issues.apache.org/jira/browse/SPARK-60948) tracks.
    - A new unit test checks the `WithRelations` relation, with a plan ID for 
each node.
    - A new e2e test, in Rust, checks a DataFrame that the query names twice.
    - New live tests check the formatted queries and that a keyword argument is 
not a parameter; they also pass with official PySpark 4.3.0. New offline tests 
check the formatter's output and its errors, whose cases also agree with 
official PySpark 4.3.0 against the server.
    - The lib tests and every integration test, e2e included, against the local 
server (641 passed), the build of all tests with `wasm-udf`, the offline Python 
suites (600 passed, 8 live cases skipped), and the live drop-in and Arrow 
suites (54 passed) and live cases (10 passed). Clippy reports no new 
diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #168 from HyukjinKwon/SPARK-60913-sql-kwargs.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/column.rs                   | 14 ++++
 crates/pyspark-rs/src/lib.rs                      |  1 +
 crates/pyspark-rs/src/session.rs                  | 53 +++++++++++----
 crates/spark-connect/src/plan.rs                  | 65 ++++++++++++++++++
 crates/spark-connect/src/session.rs               | 32 ++++++++-
 crates/spark-connect/tests/e2e_session_catalog.rs | 35 ++++++++++
 python/pyspark/sql/connect/sql_formatter.py       | 82 +++++++++++++++++++++++
 python/tests/test_dropin_e2e.py                   | 35 ++++++++++
 python/tests/test_dropin_offline.py               | 37 ++++++++++
 9 files changed, 339 insertions(+), 15 deletions(-)

diff --git a/crates/pyspark-rs/src/column.rs b/crates/pyspark-rs/src/column.rs
index b1ee90a..872613b 100644
--- a/crates/pyspark-rs/src/column.rs
+++ b/crates/pyspark-rs/src/column.rs
@@ -30,6 +30,20 @@ pub(crate) fn raise_pyspark(
     build().unwrap_or_else(|e| e)
 }
 
+/// The unparsed identifier of `column` if it is a plain column reference, 
such as `col("a")`
+/// or `df.a`, and None for another expression. `SparkSession.sql` formats 
such a column in a
+/// query as its name, as PySpark's formatter does a `ColumnReference`. A 
function rather than
+/// an attribute, as a Column's attributes are its fields in PySpark.
+#[pyfunction]
+pub(crate) fn column_reference_name(column: PyRef<'_, PyColumn>) -> 
Option<String> {
+    match column.column.expression().node() {
+        spark_connect::expression::ExpressionNode::ColumnReference(reference) 
=> {
+            Some(reference.name.clone())
+        }
+        _ => None,
+    }
+}
+
 /// Python wrapper for a Spark Column.
 #[pyclass(name = "Column", module = "pyspark.sql.connect.column")]
 pub struct PyColumn {
diff --git a/crates/pyspark-rs/src/lib.rs b/crates/pyspark-rs/src/lib.rs
index e5b3c6f..1579617 100644
--- a/crates/pyspark-rs/src/lib.rs
+++ b/crates/pyspark-rs/src/lib.rs
@@ -128,6 +128,7 @@ fn _pyspark(_py: Python<'_>, m: &Bound<'_, PyModule>) -> 
PyResult<()> {
 
     // The core DataType JSON grammar, which `pyspark.sql.types` parses type 
JSON with.
     m.add_function(wrap_pyfunction!(types::parse_datatype_json, m)?)?;
+    m.add_function(wrap_pyfunction!(column::column_reference_name, m)?)?;
 
     // Spark Declarative Pipelines (SDP) command execution
     m.add_class::<pipelines::PyPipelineRunStream>()?;
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 1432ff3..4b9ce5c 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -644,12 +644,17 @@ impl PySparkSession {
         Ok(PyDataFrame::new(df))
     }
 
-    /// Execute a SQL query and return a DataFrame. `args` binds parameters: a 
list fills
-    /// positional `?` parameters, a dict fills named (`:name`) parameters. 
Values are always
-    /// literals (a string arg is a string literal, never a column reference).
+    /// Execute a SQL query and return a DataFrame, as PySpark Connect does. 
`args` binds
+    /// parameters: a list fills positional `?` parameters and a dict named 
`:name` ones, each
+    /// value as a literal (a string arg is a string literal, never a column 
reference).
+    ///
+    /// Keyword arguments format the query, as `str.format` does, with the 
formatter of
+    /// `pyspark.sql.connect.sql_formatter`: a DataFrame becomes the name of a 
reference that
+    /// is sent with the query, a column reference its name, a string its SQL 
literal, and
+    /// another value its string.
     #[pyo3(signature = (sqlQuery, args=None, **kwargs))]
     fn sql(
-        &self,
+        slf: &Bound<'_, Self>,
         sqlQuery: &str,
         args: Option<Bound<'_, PyAny>>,
         kwargs: Option<Bound<'_, PyDict>>,
@@ -661,34 +666,54 @@ impl PySparkSession {
             }
             Ok(crate::functions::to_column(v)?.expression().clone())
         }
+        let py = slf.py();
         let mut pos: Vec<spark_connect::expression::Expression> = Vec::new();
         let mut named: std::collections::HashMap<String, 
spark_connect::expression::Expression> =
             std::collections::HashMap::new();
         if let Some(a) = args {
             if let Ok(d) = a.cast::<PyDict>() {
                 for (k, v) in d.iter() {
-                    named.insert(k.extract::<String>()?, to_lit(&v)?);
+                    // As PySpark, which asserts that a name is a string.
+                    let k = k
+                        .cast::<pyo3::types::PyString>()
+                        .map_err(|_| 
pyo3::exceptions::PyAssertionError::new_err(()))?;
+                    named.insert(k.to_cow()?.into_owned(), to_lit(&v)?);
                 }
             } else if let Ok(lst) = a.cast::<PyList>() {
                 for v in lst.iter() {
                     pos.push(to_lit(&v)?);
                 }
             } else {
-                return Err(PyErr::new::<pyo3::exceptions::PyTypeError, _>(
-                    "args must be a list (positional) or dict (named)",
+                return Err(crate::column::raise_pyspark(
+                    py,
+                    "PySparkTypeError",
+                    "INVALID_TYPE",
+                    &[
+                        ("arg_name", "args"),
+                        ("arg_type", &a.get_type().name()?.to_string()),
+                    ],
                 ));
             }
         }
-        // Named parameters may also be supplied as **kwargs (reference 
parity); merge them
-        // into the named-args map alongside any dict passed via `args`.
-        if let Some(kw) = kwargs {
-            for (k, v) in kw.iter() {
-                named.insert(k.extract::<String>()?, to_lit(&v)?);
+        let mut query = sqlQuery.to_string();
+        let mut references = Vec::new();
+        if let Some(kwargs) = kwargs.filter(|kwargs| !kwargs.is_empty()) {
+            let formatter = py
+                .import("pyspark.sql.connect.sql_formatter")?
+                .getattr("SQLStringFormatter")?
+                .call1((slf,))?;
+            query = formatter
+                .call_method("format", (sqlQuery,), Some(&kwargs))?
+                .extract()?;
+            for view in formatter.getattr("_temp_views")?.try_iter()? {
+                let (df, name): (PyRef<'_, PyDataFrame>, String) = 
view?.extract()?;
+                references.push((df.dataframe.clone(), name));
             }
         }
-        let df = self
+        let df = slf
+            .borrow()
             .session
-            .sql_with_args(sqlQuery, pos, named)
+            .sql_with_references(&query, pos, named, references)
             .to_pyerr()?;
         Ok(PyDataFrame::new(df))
     }
diff --git a/crates/spark-connect/src/plan.rs b/crates/spark-connect/src/plan.rs
index b8f0a59..b0ae87b 100644
--- a/crates/spark-connect/src/plan.rs
+++ b/crates/spark-connect/src/plan.rs
@@ -223,6 +223,13 @@ pub enum LogicalPlan {
         input: Arc<LogicalPlan>,
         alias: String,
     },
+    /// WithRelations: a plan with the plans that it refers to, such as a SQL 
query with the
+    /// DataFrames that `SparkSession.sql` takes as keyword arguments, each a 
`SubqueryAlias`
+    /// whose alias the query names.
+    WithRelations {
+        root: Arc<LogicalPlan>,
+        references: Vec<Arc<LogicalPlan>>,
+    },
     /// LocalRelation: data supplied locally.
     LocalRelation {
         schema: Option<DataType>,
@@ -1029,6 +1036,19 @@ impl LogicalPlan {
                     
Some(proto::relation::RelType::SubqueryAlias(Box::new(sq_alias)));
             }
 
+            LogicalPlan::WithRelations { root, references } => {
+                let with_relations = proto::WithRelations {
+                    root: Some(Box::new(relation_of(root, plan_ids))),
+                    references: references
+                        .iter()
+                        .map(|reference| relation_of(reference, plan_ids))
+                        .collect(),
+                };
+                relation.rel_type = 
Some(proto::relation::RelType::WithRelations(Box::new(
+                    with_relations,
+                )));
+            }
+
             LogicalPlan::ChunkedCachedLocalRelation {
                 data_hashes,
                 schema_hash,
@@ -2006,3 +2026,48 @@ mod argfix_tests {
         }
     }
 }
+
+#[cfg(test)]
+mod with_relations_tests {
+    use super::*;
+    use spark_connect_proto as proto;
+
+    /// As PySpark sends `sql()` with DataFrames: a `WithRelations` whose root 
is the query
+    /// and whose references are `SubqueryAlias`es of the DataFrames, each 
node with a plan
+    /// ID of its own.
+    #[test]
+    fn with_relations_sends_the_root_and_its_references() {
+        let ids = PlanIds::new();
+        let df = Arc::new(range(0, 3, 1));
+        let plan = Arc::new(LogicalPlan::WithRelations {
+            root: Arc::new(sql("SELECT * FROM v")),
+            references: vec![Arc::new(subquery_alias(df.clone(), "v"))],
+        });
+        let relation = relation_with_ids(&plan, &ids);
+        let Some(proto::relation::RelType::WithRelations(with)) = 
&relation.rel_type else {
+            panic!("expected WithRelations: {relation:?}");
+        };
+        let root = with.root.as_deref().expect("root");
+        assert!(
+            matches!(&root.rel_type, Some(proto::relation::RelType::Sql(s)) if 
s.query == "SELECT * FROM v")
+        );
+        assert_eq!(with.references.len(), 1);
+        let Some(proto::relation::RelType::SubqueryAlias(alias)) = 
&with.references[0].rel_type
+        else {
+            panic!("expected SubqueryAlias");
+        };
+        assert_eq!(alias.alias, "v");
+        let input = alias.input.as_deref().expect("input");
+        assert!(matches!(
+            &input.rel_type,
+            Some(proto::relation::RelType::Range(_))
+        ));
+        let id = |r: &proto::Relation| r.common.as_ref().and_then(|c| 
c.plan_id);
+        let all = [id(&relation), id(root), id(&with.references[0]), 
id(input)];
+        assert!(all.iter().all(Option::is_some), "{all:?}");
+        let distinct: std::collections::HashSet<_> = all.iter().collect();
+        assert_eq!(distinct.len(), 4, "{all:?}");
+        // The DataFrame keeps its own plan ID as a reference.
+        assert_eq!(id(input), Some(ids.id_of(&df)));
+    }
+}
diff --git a/crates/spark-connect/src/session.rs 
b/crates/spark-connect/src/session.rs
index a791e1f..96c3b1f 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -274,11 +274,41 @@ impl SparkSession {
         pos_args: Vec<crate::expression::Expression>,
         named_args: std::collections::HashMap<String, 
crate::expression::Expression>,
     ) -> Result<DataFrame> {
-        let plan = Arc::new(LogicalPlan::Sql {
+        self.sql_with_references(query, pos_args, named_args, Vec::new())
+    }
+
+    /// SQL with parameters and with DataFrames that the query names, mirroring
+    /// `SparkSession.sql(query, args, **kwargs)` with DataFrames in `kwargs`: 
each
+    /// `(DataFrame, name)` is sent along with the query, as a reference that 
the server
+    /// resolves `name` in the query to.
+    pub fn sql_with_references(
+        &self,
+        query: &str,
+        pos_args: Vec<crate::expression::Expression>,
+        named_args: std::collections::HashMap<String, 
crate::expression::Expression>,
+        references: Vec<(DataFrame, String)>,
+    ) -> Result<DataFrame> {
+        let sql = Arc::new(LogicalPlan::Sql {
             query: query.to_string(),
             pos_args,
             named_args,
         });
+        let plan = if references.is_empty() {
+            sql
+        } else {
+            Arc::new(LogicalPlan::WithRelations {
+                root: sql,
+                references: references
+                    .into_iter()
+                    .map(|(df, alias)| {
+                        Arc::new(LogicalPlan::SubqueryAlias {
+                            input: df.plan,
+                            alias,
+                        })
+                    })
+                    .collect(),
+            })
+        };
         // As pyspark, run the statement as a command now. The server executes 
a command and
         // returns a relation of its result; for a query it returns the 
query's relation, which
         // then runs at an action.
diff --git a/crates/spark-connect/tests/e2e_session_catalog.rs 
b/crates/spark-connect/tests/e2e_session_catalog.rs
index cba7b22..1e1fae8 100644
--- a/crates/spark-connect/tests/e2e_session_catalog.rs
+++ b/crates/spark-connect/tests/e2e_session_catalog.rs
@@ -257,3 +257,38 @@ fn sql_runs_statements_eagerly() {
     assert!(spark.sql("SELEC 1").is_err());
     let _ = spark.stop();
 }
+
+/// As pyspark's `sql()` with DataFrames as keyword arguments: each DataFrame 
is sent with
+/// the statement as a reference, which the server resolves the name in the 
query to, also
+/// when the query names it twice, and a command can read one too.
+#[test]
+fn sql_resolves_the_dataframes_that_the_query_names() {
+    if !should_run() {
+        return;
+    }
+    use spark_connect::row::Value;
+    use std::collections::HashMap;
+
+    let spark = session().new_session();
+    let ids = spark.range(3).expect("range");
+    let df = spark
+        .sql_with_references(
+            "SELECT count(*) AS n FROM refs_ids a JOIN refs_ids b USING (id)",
+            Vec::new(),
+            HashMap::new(),
+            vec![(ids.clone(), "refs_ids".to_string())],
+        )
+        .expect("query");
+    let rows = df.collect().expect("collect");
+    assert_eq!(rows[0].get(0), Some(&Value::Long(3)));
+    let shown = spark
+        .sql_with_references(
+            "SELECT * FROM refs_ids WHERE id > 0",
+            Vec::new(),
+            HashMap::new(),
+            vec![(ids, "refs_ids".to_string())],
+        )
+        .expect("filter");
+    assert_eq!(shown.collect().expect("collect").len(), 2);
+    let _ = spark.stop();
+}
diff --git a/python/pyspark/sql/connect/sql_formatter.py 
b/python/pyspark/sql/connect/sql_formatter.py
new file mode 100644
index 0000000..10b2c52
--- /dev/null
+++ b/python/pyspark/sql/connect/sql_formatter.py
@@ -0,0 +1,82 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""The formatting of ``SparkSession.sql`` queries with keyword arguments.
+
+Mirrors ``pyspark.sql.connect.sql_formatter`` of PySpark 4.3.0. The 
Rust-backed Column has
+no expression tree in Python, so ``column_reference_name`` of the extension 
tells whether
+it is a plain column reference, and its name.
+"""
+
+import string
+import typing
+from typing import Any, List, Mapping, Optional, Sequence, Tuple
+import uuid
+
+from pyspark.errors import PySparkValueError
+
+if typing.TYPE_CHECKING:
+    from pyspark.sql.connect.session import SparkSession
+    from pyspark.sql.connect.dataframe import DataFrame
+
+
+class SQLStringFormatter(string.Formatter):
+    """
+    A standard ``string.Formatter`` in Python that can understand PySpark 
instances
+    with basic Python objects. This object has to be clear after the use for 
single SQL
+    query; cannot be reused across multiple SQL queries without cleaning.
+    """
+
+    def __init__(self, session: "SparkSession") -> None:
+        self._session: "SparkSession" = session
+        self._temp_views: List[Tuple["DataFrame", str]] = []
+
+    def get_field(self, field_name: str, args: Sequence[Any], kwargs: 
Mapping[str, Any]) -> Any:
+        obj, first = super().get_field(field_name, args, kwargs)
+        return self._convert_value(obj, field_name), first
+
+    def _convert_value(self, val: Any, field_name: str) -> Optional[str]:
+        """
+        Converts the given value into a SQL string.
+        """
+        from pyspark._pyspark import column_reference_name
+        from pyspark.sql.connect.dataframe import DataFrame
+        from pyspark.sql.connect.column import Column
+        from pyspark.sql.utils import get_lit_sql_str
+
+        if isinstance(val, Column):
+            name = column_reference_name(val)
+            if name is not None:
+                return name
+            else:
+                raise PySparkValueError(
+                    "%s in %s should be a plain column reference such as 
`df.col` "
+                    "or `col('column')`" % (val, field_name)
+                )
+        elif isinstance(val, DataFrame):
+            for df, n in self._temp_views:
+                if df is val:
+                    return n
+            name = "_pyspark_connect_temp_view_%s" % 
str(uuid.uuid4()).replace("-", "")
+            self._temp_views.append((val, name))
+            return name
+        elif isinstance(val, str):
+            return get_lit_sql_str(val)
+        else:
+            return val
+
+    def clear(self) -> None:
+        pass
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index d6ec5e0..fc5c388 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -363,6 +363,41 @@ def 
test_sql_fails_at_a_statement_that_does_not_parse(session):
         session.sql("SELEC 1")
 
 
+def test_sql_formats_the_query_with_keyword_arguments_as_pyspark(session):
+    from decimal import Decimal
+
+    df = session.range(2)
+    other = session.createDataFrame([(0, "a"), (1, "b")], "id long, s string")
+    cases = [
+        # A value becomes its string, a string its SQL literal, and a column 
reference its
+        # name, as `str.format` with PySpark's formatter does.
+        (lambda: session.sql("SELECT {x} AS v", x=1), [(1,)]),
+        (lambda: session.sql("SELECT {s} AS s", s="it's a \\ test"), [("it's a 
\\ test",)]),
+        (lambda: session.sql("SELECT {c} AS c FROM {df}", c=F.col("id"), 
df=df), [(0,), (1,)]),
+        (lambda: session.sql("SELECT {df.id} AS c FROM {df}", df=df), [(0,), 
(1,)]),
+        (lambda: session.sql("SELECT {x:.2f} AS v", x=1.5), 
[(Decimal("1.50"),)]),
+        # A DataFrame is a reference sent with the query, one for each 
DataFrame.
+        (lambda: session.sql("SELECT count(*) AS n FROM {df}", 
df=session.range(3)), [(3,)]),
+        (lambda: session.sql("SELECT * FROM {a} UNION ALL SELECT * FROM {a}", 
a=df),
+         [(0,), (1,), (0,), (1,)]),
+        (lambda: session.sql(
+            "SELECT a.id, b.s FROM {a} a JOIN {b} b ON a.id = b.id ORDER BY 
a.id", a=df, b=other),
+         [(0, "a"), (1, "b")]),
+        # `args` still binds parameters, and braces are only formatted with 
keyword arguments.
+        (lambda: session.sql("SELECT {x} + :p AS v", args={"p": 1}, x=2), 
[(3,)]),
+        (lambda: session.sql("SELECT '{{}}' AS v"), [("{{}}",)]),
+        (lambda: session.sql("SELECT '{{}}' AS v, {x} AS x", x=1), [("{}", 
1)]),
+    ]
+    for run, rows in cases:
+        assert [tuple(r) for r in run().collect()] == rows
+
+
+def test_sql_keyword_arguments_are_not_parameters_as_pyspark(session):
+    with pytest.raises(AnalysisException) as error:
+        session.sql("SELECT :p AS v", p=5).collect()
+    assert error.value.getCondition() == "UNBOUND_SQL_PARAMETER"
+
+
 def test_maps_keep_their_entries_in_order_with_keys_of_their_type(session):
     row = session.range(1).selectExpr(
         "map(2, 'b', 1, 'a') AS ints",
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 8422e36..39e0638 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -5070,3 +5070,40 @@ def test_print_schema_udt_class_name(mock_connect, 
capsys, python_class, expecte
     assert capsys.readouterr().out == f"root\n |-- features: {expected} 
(nullable = true)\n\n"
     assert service.calls == 1
     assert service.execute_calls == 0
+
+
+# --------------------------------------------------------------------------- 
sql kwargs
+
+
+def test_sql_formatter_formats_values_as_pyspark():
+    from pyspark.sql import functions as F
+    from pyspark.sql.connect.sql_formatter import SQLStringFormatter
+
+    spark = _offline_session()
+    df = spark.range(1)
+    formatter = SQLStringFormatter(spark)
+    query = formatter.format(
+        "SELECT {c}, {d.id}, {s}, {n} FROM {df} JOIN {df}",
+        c=F.col("id"), d=df, s="it's \\", n=1, df=df)
+    # A DataFrame named twice is one reference.
+    [(view, name)] = formatter._temp_views
+    assert view is df and name.startswith("_pyspark_connect_temp_view_")
+    assert query == f"SELECT id, id, 'it\\'s \\\\', 1 FROM {name} JOIN {name}"
+
+
+def test_sql_keyword_argument_errors_as_pyspark():
+    from pyspark.errors import PySparkTypeError, PySparkValueError
+    from pyspark.sql import functions as F
+
+    # Raised before the statement is sent, as in PySpark.
+    spark = _offline_session()
+    with pytest.raises(IndexError):
+        spark.sql("SELECT {} AS v", x=1)
+    with pytest.raises(KeyError):
+        spark.sql("SELECT {y} AS v", x=1)
+    with pytest.raises(PySparkValueError, match="in c should be a plain column 
reference"):
+        spark.sql("SELECT {c}", c=F.col("id") + 1)
+    with pytest.raises(PySparkTypeError) as error:
+        spark.sql("SELECT ? AS v", args=(1,))
+    assert error.value.getCondition() == "INVALID_TYPE"
+    assert error.value.getMessageParameters() == {"arg_name": "args", 
"arg_type": "tuple"}


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to