This is an automated email from the ASF dual-hosted git repository.
gaogaotiantian pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 4f540c08607d [SPARK-60159][PYTHON][TEST] Add tests for string and
DataFrame keyword arguments in spark.sql formatting
4f540c08607d is described below
commit 4f540c08607d32d8d6512910074a8408b799659c
Author: Max <[email protected]>
AuthorDate: Sun Oct 11 01:42:19 2026 -0700
[SPARK-60159][PYTHON][TEST] Add tests for string and DataFrame keyword
arguments in spark.sql formatting
### What changes were proposed in this pull request?
This PR adds tests to `SQLTestsMixin` in
`python/pyspark/sql/tests/test_sql.py` for code paths of `spark.sql()`
keyword-argument formatting that were not covered. Since the mixin is shared by
`test_parity_sql.py`, the tests run in both classic and Spark Connect:
- `test_kwargs_string_literal`: string values round-trip unchanged,
including single and double quotes, backslashes (e.g. `C:\new`, a trailing
`\`), `\'`, control characters, an injection-like value (`x' OR '1'='1`), and
braces
- `test_kwargs_dataframe_referenced_twice`:
- the same DataFrame referenced twice by one name reuses one view
- the same DataFrame passed under two names reuses one view
- different DataFrames get different views
### Why are the changes needed?
To improve test coverage of
`pyspark.sql.connect.sql_formatter.SQLStringFormatter`. The string-literal
branch and the view-reuse branch had no tests, so regressions in
quoting/escaping or in view naming would not be caught.
### Does this PR introduce _any_ user-facing change?
No. Test-only change.
### How was this patch tested?
Added unit tests in `python/pyspark/sql/tests/test_sql.py`, run in classic
and Spark Connect:
```
python/run-tests --python-executables=python --testnames
"pyspark.sql.tests.test_sql,pyspark.sql.tests.connect.test_parity_sql"
```
Ran the Connect parity tests with coverage before and after the change:
```
python/run-tests-with-coverage --python-executables=python --testnames
pyspark.sql.tests.connect.test_parity_sql
```
Coverage of `pyspark/sql/connect/sql_formatter.py`:
| | Stmts | Miss | BrPart | Cover |
|---|---|---|---|---|
| Before | 34 | 5 | 3 | 78% |
| After | 34 | 2 | 1 | 93% |
The remaining uncovered lines are the no-op `clear()` and the error for a
non-plain column reference. The latter raises `PySparkValueError` without an
error class in Spark Connect, unlike classic
(`VALUE_NOT_PLAIN_COLUMN_REFERENCE`), so it is left out of this test-only PR.
All tests pass. `dev/reformat-python` and `dev/lint-python` pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5 (Anthropic), used as a pair-programming
assistant. The author reviewed every line, ran the tests, and verified the
coverage results locally.
Closes #59361 from LearnFirst/SPARK-60159.
Authored-by: Max <[email protected]>
Signed-off-by: Tian Gao <[email protected]>
(cherry picked from commit e81f9af4d7eb45d922cb4b6fb87eb2a2d1ffe9c6)
Signed-off-by: Tian Gao <[email protected]>
---
python/pyspark/sql/tests/test_sql.py | 52 ++++++++++++++++++++++++++++++++++++
1 file changed, 52 insertions(+)
diff --git a/python/pyspark/sql/tests/test_sql.py
b/python/pyspark/sql/tests/test_sql.py
index 1e7a3898e7b5..4c93d1fcd4fd 100644
--- a/python/pyspark/sql/tests/test_sql.py
+++ b/python/pyspark/sql/tests/test_sql.py
@@ -113,6 +113,58 @@ class SQLTestsMixin:
self.assertEqual(df1.take(1), [Row(id=5)])
self.assertEqual(df1.tail(1), [Row(id=8)])
+ def test_kwargs_string_literal(self):
+ # String values are converted to SQL string literals with quotes and
+ # backslashes escaped, so they round-trip unchanged.
+ for s in [
+ "abc",
+ "",
+ "it's",
+ 'say "hi"',
+ "back\\slash",
+ "C:\\new",
+ "end\\",
+ "\\'",
+ "x' OR '1'='1",
+ "line1\nline2\ttab",
+ "{not_a_field}",
+ ]:
+ with self.subTest(s=s):
+ self.assertEqual(self.spark.sql("SELECT {s} AS v",
s=s).first(), Row(v=s))
+
+ df = self.spark.sql(
+ "SELECT * FROM VALUES ('a'), ('b'), ('it''s') AS t(name) WHERE
name = {name}",
+ name="it's",
+ )
+ self.assertEqual(df.collect(), [Row(name="it's")])
+
+ def test_kwargs_dataframe_referenced_twice(self):
+ df = self.spark.range(3)
+ expected = [Row(a=0, b=0), Row(a=1, b=1), Row(a=2, b=2)]
+
+ # The same DataFrame referenced twice by the same name.
+ df1 = self.spark.sql(
+ "SELECT l.id AS a, r.id AS b FROM {df} l JOIN {df} r ON l.id =
r.id ORDER BY a",
+ df=df,
+ )
+ self.assertEqual(df1.collect(), expected)
+
+ # The same DataFrame passed under two different names.
+ df2 = self.spark.sql(
+ "SELECT l.id AS a, r.id AS b FROM {x} l JOIN {y} r ON l.id = r.id
ORDER BY a",
+ x=df,
+ y=df,
+ )
+ self.assertEqual(df2.collect(), expected)
+
+ # Different DataFrames must not share a name.
+ df3 = self.spark.sql(
+ "SELECT l.id AS a, r.id AS b FROM {x} l JOIN {y} r ON l.id + 1 =
r.id ORDER BY a",
+ x=df,
+ y=self.spark.range(1, 3),
+ )
+ self.assertEqual(df3.collect(), [Row(a=0, b=1), Row(a=1, b=2)])
+
def test_nested_view(self):
with self.temp_view("v1", "v2", "v3", "v4"):
self.spark.range(10).createOrReplaceTempView("v1")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
