Aditya Ranjan created SPARK-60865:
-------------------------------------
Summary: assertDataFrameEqual fails for unordered rows with
floating-point tolerance
Key: SPARK-60865
URL: https://issues.apache.org/jira/browse/SPARK-60865
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 4.2.0
Reporter: Aditya Ranjan
h3. Problem
{{pyspark.testing.assertDataFrameEqual}} can incorrectly report unequal rows
when {{checkRowOrder=False}} and floating-point values compare equal within the
configured {{rtol}} and {{atol}} tolerances.
The current implementation sorts rows using their string representations and
compares the sorted rows positionally. Lexicographic string ordering does not
preserve tolerance-aware numerical equality.
Consequently, two collections of rows can have a valid one-to-one matching
while the positional comparison fails.
h3. Steps to Reproduce
{{from pyspark.sql import Row
from pyspark.testing import assertDataFrameEqual
actual = [
Row(value=0.0),
Row(value=0.1),
]
expected = [
Row(value=0.1),
Row(value=0.2),
]
assertDataFrameEqual(
actual,
expected,
rtol=0,
atol=0.11,
checkRowOrder=False,
)}}
h3. Actual Behavior
The assertion can fail even though every actual row can be matched uniquely to
an expected row within the configured absolute tolerance.
h3. Expected Behavior
When {{{}checkRowOrder=False{}}}, the assertion should succeed if a complete
one-to-one matching exists between the actual and expected rows under the
configured comparison tolerances.
When {{{}checkRowOrder=True{}}}, positional comparison should remain enforced.
Genuinely unequal rows must continue to fail.
h3. Root Cause
Sorting rows by {{str(row)}} followed by positional comparison is not
sufficient for unordered, tolerance-aware comparisons.
A greedy matching strategy can also fail when tolerance ranges overlap and an
earlier match needs reassignment.
h3. Proposed Fix
Retain the existing deterministic sorting and fast positional comparison.
If positional comparison fails for equally sized unordered collections, attempt
one-to-one row matching using the existing tolerance-aware comparison logic.
Allow previously matched rows to be reassigned when necessary to obtain a
complete matching.
h3. Testing
The proposed implementation includes regression tests for:
* Approximate floating-point equality across unordered rows.
* Overlapping tolerance ranges requiring reassignment.
* Ordered comparisons that must detect differences.
* Unequal rows that must continue to fail.
Local validation included seven focused tests, randomized and exhaustive
matching checks, Python compilation, and Git whitespace validation.
h3. References
Original GitHub issue: [https://github.com/apache/spark/issues/60027]
Implementation PR: [https://github.com/apache/spark/pull/60047]
Note: GitHub issue #60027 is distinct from Jira SPARK-60027, which tracks an
unrelated Spark SQL issue.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
