Xiaoxuan Li created SPARK-56089:
-----------------------------------
Summary: Align asinh/acosh with fdlibm algorithm for cross-engine
compatibility
Key: SPARK-56089
URL: https://issues.apache.org/jira/browse/SPARK-56089
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.1.1, 3.5.7
Reporter: Xiaoxuan Li
The fdlibm algorithm is used across the industry:
Directly implements fdlibm:
- glibc / musl (C, the original fdlibm s_asinh.c / e_acosh.c)
- OpenJDK StrictMath/Math (Java port, FdLibm.java, JDK-8376665)
- Go standard library (ported from FreeBSD s_asinh.c)
Delegates to platform libm (which is typically fdlibm-based):
- Python math module
- PostgreSQL
- DuckDB
The fdlibm algorithm splits the computation into range-specific branches (log1p
for values near 1/0, identity for tiny values, etc.) which produces slightly
different results (1-2 ULP) compared to the naive formula. Adopting it aligns
Spark's results with these platforms, reducing inconsistencies when migrating
workloads or comparing outputs across systems.
Note: OpenJDK added StrictMath.acosh in JDK 27 (JDK-8376665) and
StrictMath.asinh in an earlier release. Once Spark upgrades its minimum JDK
version to 27+, this custom implementation can be replaced with direct calls to
StrictMath.asinh/StrictMath.acosh.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
