Akshay Thorat created SPARK-59514:
-------------------------------------
Summary: assertDataFrameEqual accepts unequal NaN and infinity
values
Key: SPARK-59514
URL: https://issues.apache.org/jira/browse/SPARK-59514
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 5.0.0
Reporter: Akshay Thorat
h2. Problem
The floating-point tolerance inequality does not reject NaN arithmetic. An
infinite expected value can also make both sides infinity. As a result, unequal
values pass equality assertions.
h2. Reproduction
Reproduced on upstream master 39776477a3d (PySpark 5.0.0.dev0), Python 3.10.11.
No SparkSession is required.
{code:python}
from pyspark.sql import Row
from pyspark.testing import assertDataFrameEqual
assertDataFrameEqual([Row(x=float("nan"))], [Row(x=1.0)])
assertDataFrameEqual([Row(x=1.0)], [Row(x=float("inf"))])
assertDataFrameEqual([Row(x=float("inf"))], [Row(x=float("-inf"))])
# All three return without raising
{code}
h2. Expected behavior
Each assertion should raise PySparkAssertionError with condition
DIFFERENT_ROWS. Matching NaNs and same-sign infinities should continue to
compare equal.
h2. Proposed fix and verification
Compare NaN and infinity explicitly before applying finite-value tolerances.
Tests cover both argument directions, zero/default tolerances, row-order
settings, and nested values. Regression tests fail before the fix and pass
afterward. Python compilation, custom-error checks and Ruff checks pass.
Pull request: https://github.com/apache/spark/pull/58790
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
