This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 014674e [SPARK-60913][CONNECT] Format spark.sql() queries with
keyword arguments as PySpark does
014674e is described below
commit 014674e44470001f5c184846964c131d43f6e4fa
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 08:46:32 2026 +0900
[SPARK-60913][CONNECT] Format spark.sql() queries with keyword arguments as
PySpark does
### What changes were proposed in this pull request?
[SPARK-60913](https://issues.apache.org/jira/browse/SPARK-60913): the
keyword arguments of `SparkSession.sql(sqlQuery, args=None, **kwargs)` now
format the query, as in PySpark Connect.
- The query is formatted as `str.format` does, with the formatter of
`pyspark.sql.connect.sql_formatter`, a shim that mirrors PySpark's:
- A DataFrame becomes a generated name,
`_pyspark_connect_temp_view_<uuid>`, one for each DataFrame.
- A plain column reference, such as `col("id")` or `df.id`, becomes its
name. Another expression raises `PySparkValueError`.
- A string becomes its SQL literal, and another value its string.
- Each DataFrame is sent with the statement as a reference that the server
resolves its name to. The statement's relation is then a `WithRelations` of the
query with a `SubqueryAlias` of each DataFrame, as in PySpark.
- `args` still binds parameters, and now raises `INVALID_TYPE` when it is
neither a dict nor a list, as in PySpark.
- In the Rust core:
- `LogicalPlan::WithRelations { root, references }` is sent as the
`with_relations` relation.
- `SparkSession::sql_with_references` takes the `(DataFrame, name)` pairs
that the query names, and `sql_with_args` calls it without any.
- In the Python client, `pyspark._pyspark.column_reference_name` tells the
formatter whether a Column is a plain column reference, and gives its name. It
is a function rather than an attribute, as the attributes of a PySpark Column
are its fields
([SPARK-60949](https://issues.apache.org/jira/browse/SPARK-60949)).
### Why are the changes needed?
The keyword arguments were bound as named parameters, and the query was
sent as is:
- `spark.sql("SELECT {x} AS v", x=1)` failed with `PARSE_SYNTAX_ERROR`,
rather than returning 1.
- `spark.sql("SELECT count(*) AS n FROM {df}", df=spark.range(3))` raised
`TypeError`, rather than returning 3.
- `spark.sql("SELECT :p AS v", p=5)` returned 5, where PySpark fails with
`UNBOUND_SQL_PARAMETER`, as `p` is not a parameter.
### Does this PR introduce _any_ user-facing change?
Yes, `sql()` with keyword arguments now behaves as in PySpark Connect
4.3.0. A keyword argument is no longer a named parameter; `args` binds those.
### How was this patch tested?
- Against a local Spark 4.3.0 server, a probe of nineteen cases gives
output identical to official PySpark 4.3.0, apart from one. The cases cover:
- values, strings with quotes and backslashes, `col()`, `df.col`, and
format specs;
- one DataFrame named twice, and a join of two;
- `args` together with keyword arguments, and `args` as a tuple;
- escaped braces with and without keyword arguments, and a command that
reads a DataFrame;
- the formatter's own errors.
The one difference is the text of the Column in the error for an
expression that is not a column reference: `Column<'(id + 1)'>` rather than
`Column<'+(id, 1)'>`, which
[SPARK-60948](https://issues.apache.org/jira/browse/SPARK-60948) tracks.
- A new unit test checks the `WithRelations` relation, with a plan ID for
each node.
- A new e2e test, in Rust, checks a DataFrame that the query names twice.
- New live tests check the formatted queries and that a keyword argument is
not a parameter; they also pass with official PySpark 4.3.0. New offline tests
check the formatter's output and its errors, whose cases also agree with
official PySpark 4.3.0 against the server.
- The lib tests and every integration test, e2e included, against the local
server (641 passed), the build of all tests with `wasm-udf`, the offline Python
suites (600 passed, 8 live cases skipped), and the live drop-in and Arrow
suites (54 passed) and live cases (10 passed). Clippy reports no new
diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #168 from HyukjinKwon/SPARK-60913-sql-kwargs.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/column.rs | 14 ++++
crates/pyspark-rs/src/lib.rs | 1 +
crates/pyspark-rs/src/session.rs | 53 +++++++++++----
crates/spark-connect/src/plan.rs | 65 ++++++++++++++++++
crates/spark-connect/src/session.rs | 32 ++++++++-
crates/spark-connect/tests/e2e_session_catalog.rs | 35 ++++++++++
python/pyspark/sql/connect/sql_formatter.py | 82 +++++++++++++++++++++++
python/tests/test_dropin_e2e.py | 35 ++++++++++
python/tests/test_dropin_offline.py | 37 ++++++++++
9 files changed, 339 insertions(+), 15 deletions(-)
diff --git a/crates/pyspark-rs/src/column.rs b/crates/pyspark-rs/src/column.rs
index b1ee90a..872613b 100644
--- a/crates/pyspark-rs/src/column.rs
+++ b/crates/pyspark-rs/src/column.rs
@@ -30,6 +30,20 @@ pub(crate) fn raise_pyspark(
build().unwrap_or_else(|e| e)
}
+/// The unparsed identifier of `column` if it is a plain column reference,
such as `col("a")`
+/// or `df.a`, and None for another expression. `SparkSession.sql` formats
such a column in a
+/// query as its name, as PySpark's formatter does a `ColumnReference`. A
function rather than
+/// an attribute, as a Column's attributes are its fields in PySpark.
+#[pyfunction]
+pub(crate) fn column_reference_name(column: PyRef<'_, PyColumn>) ->
Option<String> {
+ match column.column.expression().node() {
+ spark_connect::expression::ExpressionNode::ColumnReference(reference)
=> {
+ Some(reference.name.clone())
+ }
+ _ => None,
+ }
+}
+
/// Python wrapper for a Spark Column.
#[pyclass(name = "Column", module = "pyspark.sql.connect.column")]
pub struct PyColumn {
diff --git a/crates/pyspark-rs/src/lib.rs b/crates/pyspark-rs/src/lib.rs
index e5b3c6f..1579617 100644
--- a/crates/pyspark-rs/src/lib.rs
+++ b/crates/pyspark-rs/src/lib.rs
@@ -128,6 +128,7 @@ fn _pyspark(_py: Python<'_>, m: &Bound<'_, PyModule>) ->
PyResult<()> {
// The core DataType JSON grammar, which `pyspark.sql.types` parses type
JSON with.
m.add_function(wrap_pyfunction!(types::parse_datatype_json, m)?)?;
+ m.add_function(wrap_pyfunction!(column::column_reference_name, m)?)?;
// Spark Declarative Pipelines (SDP) command execution
m.add_class::<pipelines::PyPipelineRunStream>()?;
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 1432ff3..4b9ce5c 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -644,12 +644,17 @@ impl PySparkSession {
Ok(PyDataFrame::new(df))
}
- /// Execute a SQL query and return a DataFrame. `args` binds parameters: a
list fills
- /// positional `?` parameters, a dict fills named (`:name`) parameters.
Values are always
- /// literals (a string arg is a string literal, never a column reference).
+ /// Execute a SQL query and return a DataFrame, as PySpark Connect does.
`args` binds
+ /// parameters: a list fills positional `?` parameters and a dict named
`:name` ones, each
+ /// value as a literal (a string arg is a string literal, never a column
reference).
+ ///
+ /// Keyword arguments format the query, as `str.format` does, with the
formatter of
+ /// `pyspark.sql.connect.sql_formatter`: a DataFrame becomes the name of a
reference that
+ /// is sent with the query, a column reference its name, a string its SQL
literal, and
+ /// another value its string.
#[pyo3(signature = (sqlQuery, args=None, **kwargs))]
fn sql(
- &self,
+ slf: &Bound<'_, Self>,
sqlQuery: &str,
args: Option<Bound<'_, PyAny>>,
kwargs: Option<Bound<'_, PyDict>>,
@@ -661,34 +666,54 @@ impl PySparkSession {
}
Ok(crate::functions::to_column(v)?.expression().clone())
}
+ let py = slf.py();
let mut pos: Vec<spark_connect::expression::Expression> = Vec::new();
let mut named: std::collections::HashMap<String,
spark_connect::expression::Expression> =
std::collections::HashMap::new();
if let Some(a) = args {
if let Ok(d) = a.cast::<PyDict>() {
for (k, v) in d.iter() {
- named.insert(k.extract::<String>()?, to_lit(&v)?);
+ // As PySpark, which asserts that a name is a string.
+ let k = k
+ .cast::<pyo3::types::PyString>()
+ .map_err(|_|
pyo3::exceptions::PyAssertionError::new_err(()))?;
+ named.insert(k.to_cow()?.into_owned(), to_lit(&v)?);
}
} else if let Ok(lst) = a.cast::<PyList>() {
for v in lst.iter() {
pos.push(to_lit(&v)?);
}
} else {
- return Err(PyErr::new::<pyo3::exceptions::PyTypeError, _>(
- "args must be a list (positional) or dict (named)",
+ return Err(crate::column::raise_pyspark(
+ py,
+ "PySparkTypeError",
+ "INVALID_TYPE",
+ &[
+ ("arg_name", "args"),
+ ("arg_type", &a.get_type().name()?.to_string()),
+ ],
));
}
}
- // Named parameters may also be supplied as **kwargs (reference
parity); merge them
- // into the named-args map alongside any dict passed via `args`.
- if let Some(kw) = kwargs {
- for (k, v) in kw.iter() {
- named.insert(k.extract::<String>()?, to_lit(&v)?);
+ let mut query = sqlQuery.to_string();
+ let mut references = Vec::new();
+ if let Some(kwargs) = kwargs.filter(|kwargs| !kwargs.is_empty()) {
+ let formatter = py
+ .import("pyspark.sql.connect.sql_formatter")?
+ .getattr("SQLStringFormatter")?
+ .call1((slf,))?;
+ query = formatter
+ .call_method("format", (sqlQuery,), Some(&kwargs))?
+ .extract()?;
+ for view in formatter.getattr("_temp_views")?.try_iter()? {
+ let (df, name): (PyRef<'_, PyDataFrame>, String) =
view?.extract()?;
+ references.push((df.dataframe.clone(), name));
}
}
- let df = self
+ let df = slf
+ .borrow()
.session
- .sql_with_args(sqlQuery, pos, named)
+ .sql_with_references(&query, pos, named, references)
.to_pyerr()?;
Ok(PyDataFrame::new(df))
}
diff --git a/crates/spark-connect/src/plan.rs b/crates/spark-connect/src/plan.rs
index b8f0a59..b0ae87b 100644
--- a/crates/spark-connect/src/plan.rs
+++ b/crates/spark-connect/src/plan.rs
@@ -223,6 +223,13 @@ pub enum LogicalPlan {
input: Arc<LogicalPlan>,
alias: String,
},
+ /// WithRelations: a plan with the plans that it refers to, such as a SQL
query with the
+ /// DataFrames that `SparkSession.sql` takes as keyword arguments, each a
`SubqueryAlias`
+ /// whose alias the query names.
+ WithRelations {
+ root: Arc<LogicalPlan>,
+ references: Vec<Arc<LogicalPlan>>,
+ },
/// LocalRelation: data supplied locally.
LocalRelation {
schema: Option<DataType>,
@@ -1029,6 +1036,19 @@ impl LogicalPlan {
Some(proto::relation::RelType::SubqueryAlias(Box::new(sq_alias)));
}
+ LogicalPlan::WithRelations { root, references } => {
+ let with_relations = proto::WithRelations {
+ root: Some(Box::new(relation_of(root, plan_ids))),
+ references: references
+ .iter()
+ .map(|reference| relation_of(reference, plan_ids))
+ .collect(),
+ };
+ relation.rel_type =
Some(proto::relation::RelType::WithRelations(Box::new(
+ with_relations,
+ )));
+ }
+
LogicalPlan::ChunkedCachedLocalRelation {
data_hashes,
schema_hash,
@@ -2006,3 +2026,48 @@ mod argfix_tests {
}
}
}
+
+#[cfg(test)]
+mod with_relations_tests {
+ use super::*;
+ use spark_connect_proto as proto;
+
+ /// As PySpark sends `sql()` with DataFrames: a `WithRelations` whose root
is the query
+ /// and whose references are `SubqueryAlias`es of the DataFrames, each
node with a plan
+ /// ID of its own.
+ #[test]
+ fn with_relations_sends_the_root_and_its_references() {
+ let ids = PlanIds::new();
+ let df = Arc::new(range(0, 3, 1));
+ let plan = Arc::new(LogicalPlan::WithRelations {
+ root: Arc::new(sql("SELECT * FROM v")),
+ references: vec![Arc::new(subquery_alias(df.clone(), "v"))],
+ });
+ let relation = relation_with_ids(&plan, &ids);
+ let Some(proto::relation::RelType::WithRelations(with)) =
&relation.rel_type else {
+ panic!("expected WithRelations: {relation:?}");
+ };
+ let root = with.root.as_deref().expect("root");
+ assert!(
+ matches!(&root.rel_type, Some(proto::relation::RelType::Sql(s)) if
s.query == "SELECT * FROM v")
+ );
+ assert_eq!(with.references.len(), 1);
+ let Some(proto::relation::RelType::SubqueryAlias(alias)) =
&with.references[0].rel_type
+ else {
+ panic!("expected SubqueryAlias");
+ };
+ assert_eq!(alias.alias, "v");
+ let input = alias.input.as_deref().expect("input");
+ assert!(matches!(
+ &input.rel_type,
+ Some(proto::relation::RelType::Range(_))
+ ));
+ let id = |r: &proto::Relation| r.common.as_ref().and_then(|c|
c.plan_id);
+ let all = [id(&relation), id(root), id(&with.references[0]),
id(input)];
+ assert!(all.iter().all(Option::is_some), "{all:?}");
+ let distinct: std::collections::HashSet<_> = all.iter().collect();
+ assert_eq!(distinct.len(), 4, "{all:?}");
+ // The DataFrame keeps its own plan ID as a reference.
+ assert_eq!(id(input), Some(ids.id_of(&df)));
+ }
+}
diff --git a/crates/spark-connect/src/session.rs
b/crates/spark-connect/src/session.rs
index a791e1f..96c3b1f 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -274,11 +274,41 @@ impl SparkSession {
pos_args: Vec<crate::expression::Expression>,
named_args: std::collections::HashMap<String,
crate::expression::Expression>,
) -> Result<DataFrame> {
- let plan = Arc::new(LogicalPlan::Sql {
+ self.sql_with_references(query, pos_args, named_args, Vec::new())
+ }
+
+ /// SQL with parameters and with DataFrames that the query names, mirroring
+ /// `SparkSession.sql(query, args, **kwargs)` with DataFrames in `kwargs`:
each
+ /// `(DataFrame, name)` is sent along with the query, as a reference that
the server
+ /// resolves `name` in the query to.
+ pub fn sql_with_references(
+ &self,
+ query: &str,
+ pos_args: Vec<crate::expression::Expression>,
+ named_args: std::collections::HashMap<String,
crate::expression::Expression>,
+ references: Vec<(DataFrame, String)>,
+ ) -> Result<DataFrame> {
+ let sql = Arc::new(LogicalPlan::Sql {
query: query.to_string(),
pos_args,
named_args,
});
+ let plan = if references.is_empty() {
+ sql
+ } else {
+ Arc::new(LogicalPlan::WithRelations {
+ root: sql,
+ references: references
+ .into_iter()
+ .map(|(df, alias)| {
+ Arc::new(LogicalPlan::SubqueryAlias {
+ input: df.plan,
+ alias,
+ })
+ })
+ .collect(),
+ })
+ };
// As pyspark, run the statement as a command now. The server executes
a command and
// returns a relation of its result; for a query it returns the
query's relation, which
// then runs at an action.
diff --git a/crates/spark-connect/tests/e2e_session_catalog.rs
b/crates/spark-connect/tests/e2e_session_catalog.rs
index cba7b22..1e1fae8 100644
--- a/crates/spark-connect/tests/e2e_session_catalog.rs
+++ b/crates/spark-connect/tests/e2e_session_catalog.rs
@@ -257,3 +257,38 @@ fn sql_runs_statements_eagerly() {
assert!(spark.sql("SELEC 1").is_err());
let _ = spark.stop();
}
+
+/// As pyspark's `sql()` with DataFrames as keyword arguments: each DataFrame
is sent with
+/// the statement as a reference, which the server resolves the name in the
query to, also
+/// when the query names it twice, and a command can read one too.
+#[test]
+fn sql_resolves_the_dataframes_that_the_query_names() {
+ if !should_run() {
+ return;
+ }
+ use spark_connect::row::Value;
+ use std::collections::HashMap;
+
+ let spark = session().new_session();
+ let ids = spark.range(3).expect("range");
+ let df = spark
+ .sql_with_references(
+ "SELECT count(*) AS n FROM refs_ids a JOIN refs_ids b USING (id)",
+ Vec::new(),
+ HashMap::new(),
+ vec![(ids.clone(), "refs_ids".to_string())],
+ )
+ .expect("query");
+ let rows = df.collect().expect("collect");
+ assert_eq!(rows[0].get(0), Some(&Value::Long(3)));
+ let shown = spark
+ .sql_with_references(
+ "SELECT * FROM refs_ids WHERE id > 0",
+ Vec::new(),
+ HashMap::new(),
+ vec![(ids, "refs_ids".to_string())],
+ )
+ .expect("filter");
+ assert_eq!(shown.collect().expect("collect").len(), 2);
+ let _ = spark.stop();
+}
diff --git a/python/pyspark/sql/connect/sql_formatter.py
b/python/pyspark/sql/connect/sql_formatter.py
new file mode 100644
index 0000000..10b2c52
--- /dev/null
+++ b/python/pyspark/sql/connect/sql_formatter.py
@@ -0,0 +1,82 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""The formatting of ``SparkSession.sql`` queries with keyword arguments.
+
+Mirrors ``pyspark.sql.connect.sql_formatter`` of PySpark 4.3.0. The
Rust-backed Column has
+no expression tree in Python, so ``column_reference_name`` of the extension
tells whether
+it is a plain column reference, and its name.
+"""
+
+import string
+import typing
+from typing import Any, List, Mapping, Optional, Sequence, Tuple
+import uuid
+
+from pyspark.errors import PySparkValueError
+
+if typing.TYPE_CHECKING:
+ from pyspark.sql.connect.session import SparkSession
+ from pyspark.sql.connect.dataframe import DataFrame
+
+
+class SQLStringFormatter(string.Formatter):
+ """
+ A standard ``string.Formatter`` in Python that can understand PySpark
instances
+ with basic Python objects. This object has to be clear after the use for
single SQL
+ query; cannot be reused across multiple SQL queries without cleaning.
+ """
+
+ def __init__(self, session: "SparkSession") -> None:
+ self._session: "SparkSession" = session
+ self._temp_views: List[Tuple["DataFrame", str]] = []
+
+ def get_field(self, field_name: str, args: Sequence[Any], kwargs:
Mapping[str, Any]) -> Any:
+ obj, first = super().get_field(field_name, args, kwargs)
+ return self._convert_value(obj, field_name), first
+
+ def _convert_value(self, val: Any, field_name: str) -> Optional[str]:
+ """
+ Converts the given value into a SQL string.
+ """
+ from pyspark._pyspark import column_reference_name
+ from pyspark.sql.connect.dataframe import DataFrame
+ from pyspark.sql.connect.column import Column
+ from pyspark.sql.utils import get_lit_sql_str
+
+ if isinstance(val, Column):
+ name = column_reference_name(val)
+ if name is not None:
+ return name
+ else:
+ raise PySparkValueError(
+ "%s in %s should be a plain column reference such as
`df.col` "
+ "or `col('column')`" % (val, field_name)
+ )
+ elif isinstance(val, DataFrame):
+ for df, n in self._temp_views:
+ if df is val:
+ return n
+ name = "_pyspark_connect_temp_view_%s" %
str(uuid.uuid4()).replace("-", "")
+ self._temp_views.append((val, name))
+ return name
+ elif isinstance(val, str):
+ return get_lit_sql_str(val)
+ else:
+ return val
+
+ def clear(self) -> None:
+ pass
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index d6ec5e0..fc5c388 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -363,6 +363,41 @@ def
test_sql_fails_at_a_statement_that_does_not_parse(session):
session.sql("SELEC 1")
+def test_sql_formats_the_query_with_keyword_arguments_as_pyspark(session):
+ from decimal import Decimal
+
+ df = session.range(2)
+ other = session.createDataFrame([(0, "a"), (1, "b")], "id long, s string")
+ cases = [
+ # A value becomes its string, a string its SQL literal, and a column
reference its
+ # name, as `str.format` with PySpark's formatter does.
+ (lambda: session.sql("SELECT {x} AS v", x=1), [(1,)]),
+ (lambda: session.sql("SELECT {s} AS s", s="it's a \\ test"), [("it's a
\\ test",)]),
+ (lambda: session.sql("SELECT {c} AS c FROM {df}", c=F.col("id"),
df=df), [(0,), (1,)]),
+ (lambda: session.sql("SELECT {df.id} AS c FROM {df}", df=df), [(0,),
(1,)]),
+ (lambda: session.sql("SELECT {x:.2f} AS v", x=1.5),
[(Decimal("1.50"),)]),
+ # A DataFrame is a reference sent with the query, one for each
DataFrame.
+ (lambda: session.sql("SELECT count(*) AS n FROM {df}",
df=session.range(3)), [(3,)]),
+ (lambda: session.sql("SELECT * FROM {a} UNION ALL SELECT * FROM {a}",
a=df),
+ [(0,), (1,), (0,), (1,)]),
+ (lambda: session.sql(
+ "SELECT a.id, b.s FROM {a} a JOIN {b} b ON a.id = b.id ORDER BY
a.id", a=df, b=other),
+ [(0, "a"), (1, "b")]),
+ # `args` still binds parameters, and braces are only formatted with
keyword arguments.
+ (lambda: session.sql("SELECT {x} + :p AS v", args={"p": 1}, x=2),
[(3,)]),
+ (lambda: session.sql("SELECT '{{}}' AS v"), [("{{}}",)]),
+ (lambda: session.sql("SELECT '{{}}' AS v, {x} AS x", x=1), [("{}",
1)]),
+ ]
+ for run, rows in cases:
+ assert [tuple(r) for r in run().collect()] == rows
+
+
+def test_sql_keyword_arguments_are_not_parameters_as_pyspark(session):
+ with pytest.raises(AnalysisException) as error:
+ session.sql("SELECT :p AS v", p=5).collect()
+ assert error.value.getCondition() == "UNBOUND_SQL_PARAMETER"
+
+
def test_maps_keep_their_entries_in_order_with_keys_of_their_type(session):
row = session.range(1).selectExpr(
"map(2, 'b', 1, 'a') AS ints",
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 8422e36..39e0638 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -5070,3 +5070,40 @@ def test_print_schema_udt_class_name(mock_connect,
capsys, python_class, expecte
assert capsys.readouterr().out == f"root\n |-- features: {expected}
(nullable = true)\n\n"
assert service.calls == 1
assert service.execute_calls == 0
+
+
+# ---------------------------------------------------------------------------
sql kwargs
+
+
+def test_sql_formatter_formats_values_as_pyspark():
+ from pyspark.sql import functions as F
+ from pyspark.sql.connect.sql_formatter import SQLStringFormatter
+
+ spark = _offline_session()
+ df = spark.range(1)
+ formatter = SQLStringFormatter(spark)
+ query = formatter.format(
+ "SELECT {c}, {d.id}, {s}, {n} FROM {df} JOIN {df}",
+ c=F.col("id"), d=df, s="it's \\", n=1, df=df)
+ # A DataFrame named twice is one reference.
+ [(view, name)] = formatter._temp_views
+ assert view is df and name.startswith("_pyspark_connect_temp_view_")
+ assert query == f"SELECT id, id, 'it\\'s \\\\', 1 FROM {name} JOIN {name}"
+
+
+def test_sql_keyword_argument_errors_as_pyspark():
+ from pyspark.errors import PySparkTypeError, PySparkValueError
+ from pyspark.sql import functions as F
+
+ # Raised before the statement is sent, as in PySpark.
+ spark = _offline_session()
+ with pytest.raises(IndexError):
+ spark.sql("SELECT {} AS v", x=1)
+ with pytest.raises(KeyError):
+ spark.sql("SELECT {y} AS v", x=1)
+ with pytest.raises(PySparkValueError, match="in c should be a plain column
reference"):
+ spark.sql("SELECT {c}", c=F.col("id") + 1)
+ with pytest.raises(PySparkTypeError) as error:
+ spark.sql("SELECT ? AS v", args=(1,))
+ assert error.value.getCondition() == "INVALID_TYPE"
+ assert error.value.getMessageParameters() == {"arg_name": "args",
"arg_type": "tuple"}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]