laserninja opened a new pull request, #58790:
URL: https://github.com/apache/spark/pull/58790
### What changes were proposed in this pull request?
Compare NaN and infinity explicitly before applying finite-value tolerances.
Preserve matching NaNs and same-sign infinities, and add tests for both
argument directions, zero/default tolerances, and nested values.
### Why are the changes needed?
The tolerance inequality does not reject NaN arithmetic, and an infinite
expected value can make both sides infinity. As a result, NaN versus a finite
value and opposite-sign infinities can pass equality assertions.
```python
from pyspark.sql import Row
from pyspark.testing import assertDataFrameEqual
assertDataFrameEqual([Row(x=float("nan"))], [Row(x=1.0)])
assertDataFrameEqual([Row(x=1.0)], [Row(x=float("inf"))])
assertDataFrameEqual([Row(x=float("inf"))], [Row(x=float("-inf"))])
# All three return without raising
```
### Does this PR introduce _any_ user-facing change?
Yes. Unequal special values now raise DIFFERENT_ROWS. Matching NaNs and
same-sign infinities continue to compare equal, and finite-value tolerance
behavior is unchanged.
### How was this patch tested?
The new regression tests failed before the fix and passed afterward on
Python 3.10.11 with NumPy 2.2.6. The focused run passed 3 tests, including
parameterized subtests:
```bash
PYTHONPATH=python TERM=dumb python -m unittest
pyspark.sql.tests.test_utils.NonFiniteComparisonTests -q
PYTHON_EXECUTABLE=python ./dev/lint-python --compile --custom-pyspark-error
--ruff
```
Run the commands with the Python virtual environment activated; the local
run shared an environment from a sibling checkout. The tests call public Python
APIs directly and require no SparkSession. Python compilation, custom-error
checks, Ruff lint, and Ruff formatting passed. JVM-backed suites were not run
because these fixes are confined to Python-side logic.
Draft pending creation and linking of the corresponding ASF Jira issue.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
