hzmsh opened a new pull request, #59967: URL: https://github.com/apache/spark/pull/59967
### What changes were proposed in this pull request? Fixes [SPARK-60761](https://issues.apache.org/jira/browse/SPARK-60761). This PR removes `assert len(columnNames) > 0` from `WithColumns.__init__` in `python/pyspark/sql/connect/plan.py`. With it gone, `df.withColumns({})` in Spark Connect returns the DataFrame unchanged, the same as in classic PySpark. The server already handles an empty dict: the client sends it as an empty list of columns, and `SparkConnectPlanner.transformWithColumns` passes that to `Dataset.withColumns`, which keeps the same columns and rows. ### Why are the changes needed? `df.withColumns({})` works in classic PySpark but fails in Spark Connect with an `AssertionError` that has no message. An empty dict can come from building the column map in code when no columns match a filter. A user on Databricks serverless (which uses Spark Connect) hit this and worked around it with `df.select("*", *exprs)` ([forum thread](https://community.databricks.com/t5/data-engineering/bug-withcolumns-in-pyspark-doesn-t-handle-empty-dictionary/td-p/136875)): ```python # spark = SparkSession.builder.remote("local[2]").getOrCreate() for Connect, .master("local[2]") for classic from pyspark.sql import functions as F df = spark.range(3) new_cols = {c + "_x": F.col(c) for c in ["missing"] if c in df.columns} # {} df.withColumns(new_cols).columns # classic: ['id'] # Connect on master, 4.2.0 and 4.3.0: AssertionError (no message) ``` The assert was added in [SPARK-41936](https://issues.apache.org/jira/browse/SPARK-41936) (#39445), when `DataFrame.withMetadata` (which always passes one column) started reusing the `WithColumns` plan. ### Does this PR introduce _any_ user-facing change? Yes. In Spark Connect, `df.withColumns({})` now returns the DataFrame unchanged instead of raising `AssertionError`. Calls with a non-empty dict, and classic PySpark, behave as before. ### How was this patch tested? Added an empty dict case to `DataFrameTestsMixin.test_with_columns`, which runs in classic (`test_dataframe`) and Connect (`test_parity_dataframe`). Without the fix it passes in classic and fails in Connect with `AssertionError`. ``` cd python && python -m unittest \ pyspark.sql.tests.test_dataframe.DataFrameTests.test_with_columns \ pyspark.sql.tests.connect.test_parity_dataframe.DataFrameParityTests.test_with_columns dev/lint-python --mypy ruff check python/pyspark/sql/connect/plan.py python/pyspark/sql/tests/test_dataframe.py ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) Co-developed with Claude Code (Claude Opus 5.5): the fix, the test and this description. The checks above were run locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
