Yingxuan Li created SPARK-61436:
-----------------------------------

             Summary: Quote literal column names when ignoring column order in 
assertDataFrameEqual
                 Key: SPARK-61436
                 URL: https://issues.apache.org/jira/browse/SPARK-61436
             Project: Spark
          Issue Type: Bug
          Components: PySpark
    Affects Versions: 5.0.0
            Reporter: Yingxuan Li


assertDataFrameEqual(..., ignoreColumnOrder=True) selects sorted column names 
without quoting them. Legal top-level names containing dots, backticks or a 
literal * can be interpreted as expressions. This can reject equal DataFrames 
and silently accept unequal ones when a dotted top-level name overlaps a nested 
field path.

Reproducer in a Spark session:
{code:python}
from pyspark.sql import Row
from pyspark.testing import assertDataFrameEqual

actual = spark.createDataFrame([(Row(b=1), 2)], ["a", "a.b"])
expected = spark.createDataFrame([(3, Row(b=1))], ["a.b", "a"])
assertDataFrameEqual(actual, expected, ignoreColumnOrder=True)
{code}

Expected: raise PySparkAssertionError because the literal top-level column a.b 
differs (2 versus 3).
Actual before the patch: comparison succeeds because both projections read the 
nested field instead.

The proposed fix quotes and escapes each literal name before projection. Two 
regression methods cover dots, embedded backticks, a literal star, and a dotted 
top-level column coexisting with a struct field, with equal and unequal inputs.

PR: https://github.com/apache/spark/pull/60225
Source validation: 
https://github.com/LunaLi3781/spark-regression-test-optimizer/actions/runs/42515177490

Validation: Spark source build with Hive and native 
pyspark.sql.tests.test_utils passed for patch commit 
4075e4b7a914cf67e5f35ee039c37df9c12cfaec (Python 3.11 / Java 21). The 
checked-out source is 5.0.0-SNAPSHOT. Focused before/after validation also used 
PySpark 4.0.1 runtime with checked-out utility code. This does not establish 
the earliest affected release. Connect parity and official PR checks remain 
pending.

AI assistance: OpenAI Codex was used for investigation, implementation, 
regression tests and description.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to