This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 3afa7999adbb [SPARK-58580][PS][FOLLOWUP] Preserve ldexp zero and
infinity inputs
3afa7999adbb is described below
commit 3afa7999adbb6ea9f50d689ccb01ba8cfb353e3e
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Thu Aug 6 19:26:00 2026 +0800
[SPARK-58580][PS][FOLLOWUP] Preserve ldexp zero and infinity inputs
### What changes were proposed in this pull request?
This follow-up updates the native `np.ldexp` mapping to return a zero or
infinite significand directly before evaluating `pow(2.0, exp)`. This avoids
the invalid intermediate products `0.0 * Infinity` and `Infinity * 0.0`.
### Why are the changes needed?
For extreme exponents, the standalone power expression can overflow or
underflow before multiplication. The prior native mapping therefore returned
`NaN` for valid inputs such as `np.ldexp(0.0, 1024)` and `np.ldexp(np.inf,
-1075)`, instead of preserving zero or infinity as NumPy does.
### Does this PR introduce _any_ user-facing change?
Yes. It corrects `np.ldexp` results for zero and infinite significands with
exponents that overflow or underflow the power intermediate.
### How was this patch tested?
Added cases for positive and negative zero with exponent `1024`, plus
positive and negative infinity with exponent `-1075`. The test also checks
result sign bits.
- `python/run-tests --testnames
pyspark.pandas.tests.test_numpy_compat.NumPyCompatTests.test_np_ldexp`
- `python/run-tests --testnames
pyspark.pandas.tests.connect.test_parity_numpy_compat.NumPyCompatParityTests.test_np_ldexp`
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex GPT-5
Closes #57820 from zhengruifeng/pandas-native-ldexp-followup-dev2.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
---
python/pyspark/pandas/numpy_compat.py | 5 +++-
python/pyspark/pandas/tests/test_numpy_compat.py | 29 ++++++++++++++++++++++--
2 files changed, 31 insertions(+), 3 deletions(-)
diff --git a/python/pyspark/pandas/numpy_compat.py
b/python/pyspark/pandas/numpy_compat.py
index 4ca221edd8ac..33538ef9de87 100644
--- a/python/pyspark/pandas/numpy_compat.py
+++ b/python/pyspark/pandas/numpy_compat.py
@@ -146,7 +146,10 @@ binary_np_spark_mappings = {
.otherwise(F.lit(1.0)),
"hypot": F.hypot,
"lcm": pandas_udf(lambda s1, s2: np.lcm(s1, s2), DoubleType()), # type:
ignore[call-overload]
- "ldexp": lambda c1, c2: c1.cast("double") * F.pow(F.lit(2.0), c2),
+ "ldexp": lambda c1, c2: F.when(
+ c1.cast("double").isin(0.0, float("-inf"), float("inf")),
+ c1.cast("double"),
+ ).otherwise(c1.cast("double") * F.pow(F.lit(2.0), c2)),
# F.shiftleft accepts literal counts only; call_function also accepts a
column.
# NumPy returns zero for counts outside an int64's bit width, unlike JVM
shifts.
"left_shift": lambda c1, c2: F.when((c2 < 0) | (c2 >= 64),
F.lit(0)).otherwise(
diff --git a/python/pyspark/pandas/tests/test_numpy_compat.py
b/python/pyspark/pandas/tests/test_numpy_compat.py
index 0c19b8bfe47b..d95474f7fc07 100644
--- a/python/pyspark/pandas/tests/test_numpy_compat.py
+++ b/python/pyspark/pandas/tests/test_numpy_compat.py
@@ -188,13 +188,38 @@ class NumPyCompatTestsMixin:
1.0,
1.0,
1.0,
+ 0.0,
+ -0.0,
+ np.inf,
+ -np.inf,
+ ],
+ "exp": [
+ 2,
+ 3,
+ -2,
+ -3,
+ -3,
+ 0,
+ -2,
+ 2,
+ 2,
+ 2,
+ -1074,
+ -1075,
+ 1024,
+ 1024,
+ 1024,
+ -1075,
+ -1075,
],
- "exp": [2, 3, -2, -3, -3, 0, -2, 2, 2, 2, -1074, -1075, 1024],
}
)
psdf = ps.from_pandas(pdf)
- self.assert_eq(np.ldexp(psdf.x, psdf.exp), np.ldexp(pdf.x, pdf.exp),
almost=True)
+ result = np.ldexp(psdf.x, psdf.exp)
+ expected = np.ldexp(pdf.x, pdf.exp)
+ self.assert_eq(result, expected, almost=True)
+ self.assert_eq(np.signbit(result.to_pandas()), np.signbit(expected))
def test_np_fmod(self):
for pdf in (
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
