Shuheng Ma created SPARK-60761:
----------------------------------
Summary: Spark Connect DataFrame.withColumns({}) fails with an
empty AssertionError, while classic returns the DataFrame unchanged
Key: SPARK-60761
URL: https://issues.apache.org/jira/browse/SPARK-60761
Project: Spark
Issue Type: Bug
Components: Connect, PySpark
Affects Versions: 4.2.0, 4.3.0, 5.0.0
Reporter: Shuheng Ma
h3. Description
In Spark Connect, {{DataFrame.withColumns({})}} raises an {{AssertionError}}
with no message, while classic PySpark returns the DataFrame unchanged.
An empty dict usually comes from building the column map in code, for example
by filtering a list of columns where none match. A user on Databricks
serverless (which uses Spark Connect) reported this:
https://community.databricks.com/t5/data-engineering/bug-withcolumns-in-pyspark-doesn-t-handle-empty-dictionary/td-p/136875
h3. Reproducer
{code:python}
# spark = SparkSession.builder.remote("local[2]").getOrCreate() for Connect,
.master("local[2]") for classic
from pyspark.sql import functions as F
df = spark.range(3)
new_cols = {c + "_x": F.col(c) for c in ["missing"] if c in df.columns} # {}
df.withColumns(new_cols).columns
# classic: ['id']
# Connect: AssertionError (no message)
{code}
h3. Cause
{{WithColumns.__init__}} in {{python/pyspark/sql/connect/plan.py}} has {{assert
len(columnNames) > 0}}:
https://github.com/apache/spark/blob/1dc0839ca2918ae78eae39635a551648c4e0789f/python/pyspark/sql/connect/plan.py#L688
The assert was added in SPARK-41936, when {{DataFrame.withMetadata}} (which
always passes one column) started reusing the {{WithColumns}} plan. The server
side already supports an empty list:
{{SparkConnectPlanner.transformWithColumns}} passes it to
{{Dataset.withColumns}}, which returns the DataFrame unchanged.
h3. Suggested fix
Remove that assert, and add the empty dict case to the shared
{{test_with_columns}} test so both classic and Connect run it.
Tested on PySpark 4.2.0 and 4.3.0 from PyPI and on master (5.0.0-SNAPSHOT),
Python 3.12, with a local Spark Connect server (first line of the reproducer).
I have a fix with tests ready and will open a PR shortly.
Co-developed with Claude Code (Claude Opus 5.5): the investigation, the
reproducer and this description.
Generated-by: Claude Code (Claude Opus 5.5)
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
