Yingxuan Li created SPARK-61436:
-----------------------------------
Summary: Quote literal column names when ignoring column order in
assertDataFrameEqual
Key: SPARK-61436
URL: https://issues.apache.org/jira/browse/SPARK-61436
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 5.0.0
Reporter: Yingxuan Li
assertDataFrameEqual(..., ignoreColumnOrder=True) selects sorted column names
without quoting them. Legal top-level names containing dots, backticks or a
literal * can be interpreted as expressions. This can reject equal DataFrames
and silently accept unequal ones when a dotted top-level name overlaps a nested
field path.
Reproducer in a Spark session:
{code:python}
from pyspark.sql import Row
from pyspark.testing import assertDataFrameEqual
actual = spark.createDataFrame([(Row(b=1), 2)], ["a", "a.b"])
expected = spark.createDataFrame([(3, Row(b=1))], ["a.b", "a"])
assertDataFrameEqual(actual, expected, ignoreColumnOrder=True)
{code}
Expected: raise PySparkAssertionError because the literal top-level column a.b
differs (2 versus 3).
Actual before the patch: comparison succeeds because both projections read the
nested field instead.
The proposed fix quotes and escapes each literal name before projection. Two
regression methods cover dots, embedded backticks, a literal star, and a dotted
top-level column coexisting with a struct field, with equal and unequal inputs.
PR: https://github.com/apache/spark/pull/60225
Source validation:
https://github.com/LunaLi3781/spark-regression-test-optimizer/actions/runs/42515177490
Validation: Spark source build with Hive and native
pyspark.sql.tests.test_utils passed for patch commit
4075e4b7a914cf67e5f35ee039c37df9c12cfaec (Python 3.11 / Java 21). The
checked-out source is 5.0.0-SNAPSHOT. Focused before/after validation also used
PySpark 4.0.1 runtime with checked-out utility code. This does not establish
the earliest affected release. Connect parity and official PR checks remain
pending.
AI assistance: OpenAI Codex was used for investigation, implementation,
regression tests and description.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]