Max Gekk created SPARK-61175:
--------------------------------
Summary: Make rint return a canonical NaN regardless of the JIT
tier
Key: SPARK-61175
URL: https://issues.apache.org/jira/browse/SPARK-61175
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Max Gekk
rint() can return NaNs with different bit patterns for the same input,
depending on which JIT tier executes java.lang.Math.rint.
Spark's Rint calls Math.rint in both eval and generated code. On HotSpot, the
interpreter and C1 run the Java implementation (StrictMath.rint), which clears
the sign of a NaN, while C2 replaces the call with an intrinsic (roundsd on
x86) that returns the input NaN with its sign. Measured on JDK 25.0.4.1, x86_64
(AMD Zen 5):
{noformat}
input bits -Xint / C1 (TieredStopAtLevel=1) C2
(-XX:-TieredCompilation)
fff8000000000000 7ff8000000000000 fff8000000000000
fff0000000000001 7ff8000000000001 fff8000000000001
fff4000000000000 7ffc000000000000 fffc000000000000
{noformat}
-XX:+PrintIntrinsics shows Math::rint as an intrinsic under C2. Every tier
keeps the payload and quiets a signalling NaN; only the sign differs.
The bits reach user code: UnsafeRowWriter.write(double) and UnsafeRow.setDouble
store the double as is, so collect(), typed Datasets, Scala/Java/Python UDF
arguments and Arrow-based paths (toPandas with Arrow, mapInArrow) can observe
the sign, e.g. with Double.doubleToRawLongBits, Math.copySign or numpy.signbit.
Comparisons, grouping, joins, sorting, hashing and string output treat all NaNs
alike, and the Parquet and ORC writers canonicalize NaN
(Double.doubleToLongBits), so SQL results and those files are not affected.
Proposed fix: make Rint return the canonical Double.NaN when its result is NaN,
in both the interpreted and the generated path, leaving every non-NaN result
unchanged.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
