This is an automated email from the ASF dual-hosted git repository.
ptoth pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new 755b09d0b7b4 [SPARK-55975][SQL][TESTS] NaN comparison can cause false
UT failures due to different NaNs
755b09d0b7b4 is described below
commit 755b09d0b7b4be7d83f01457eaaee5aa9e008209
Author: Marco Gaido <[email protected]>
AuthorDate: Fri Mar 13 09:43:14 2026 +0100
[SPARK-55975][SQL][TESTS] NaN comparison can cause false UT failures due to
different NaNs
### What changes were proposed in this pull request?
In the UTs, currently NaN comparisons can cause UT failures on the CI
(which is also hardware dependent) as on some hardware NaN can be represented
in different byte formats (see IEEE-754 that allows multiple bit patterns for
NaN). This has caused spurious UT failures e.g. in
https://github.com/apache/spark/pull/54676.
### Why are the changes needed?
Fix NaN comparison so that we do not run into hardware-dependent UT
failures and correctly passes when there are two NaNs.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Added UT.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #54772 from mgaido91/SPARK-55975.
Authored-by: Marco Gaido <[email protected]>
Signed-off-by: Peter Toth <[email protected]>
(cherry picked from commit e7bbd323ec75e0ef96658c588146950131756f1e)
Signed-off-by: Peter Toth <[email protected]>
---
.../src/test/scala/org/apache/spark/sql/MathFunctionsSuite.scala | 9 +++++++++
sql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala | 3 +++
2 files changed, 12 insertions(+)
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/MathFunctionsSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/MathFunctionsSuite.scala
index f1d0815c181b..2e9dae6a0dee 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/MathFunctionsSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/MathFunctionsSuite.scala
@@ -117,6 +117,15 @@ class MathFunctionsSuite extends QueryTest with
SharedSparkSession {
)
}
+ test("different NaN comparison") {
+ assert(QueryTest.compare(
+ java.lang.Double.longBitsToDouble(0x7ff8000000000000L),
+ java.lang.Double.longBitsToDouble(0xfff8000000000000L)))
+ assert(QueryTest.compare(
+ java.lang.Float.intBitsToFloat(0x7fc00000),
+ java.lang.Float.intBitsToFloat(0xffc00000)))
+ }
+
test("sin") {
testOneToOneMathFunction(sin, math.sin)
}
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala
b/sql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala
index 71af79895d20..bf2c86c1f13d 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala
@@ -405,9 +405,12 @@ object QueryTest extends Assertions {
case (a: Row, b: Row) =>
compare(a.toSeq, b.toSeq)
// 0.0 == -0.0, turn float/double to bits before comparison, to
distinguish 0.0 and -0.0.
+ // in some hardware NaN can be represented with different bits, so first
check for it
case (a: Double, b: Double) =>
+ a.isNaN && b.isNaN ||
java.lang.Double.doubleToRawLongBits(a) ==
java.lang.Double.doubleToRawLongBits(b)
case (a: Float, b: Float) =>
+ a.isNaN && b.isNaN ||
java.lang.Float.floatToRawIntBits(a) ==
java.lang.Float.floatToRawIntBits(b)
case (a, b) => a == b
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
