This is an automated email from the ASF dual-hosted git repository.
Yicong-Huang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 34aad86c44b6 [SPARK-58646][PS][FOLLOWUP] Fix NumPy reciprocal parity
for decimal, boolean, and narrower integer columns
34aad86c44b6 is described below
commit 34aad86c44b68b49ab937c08bcf395dd0082a838
Author: Yicong Huang <[email protected]>
AuthorDate: Wed Aug 26 09:57:26 2026 +0000
[SPARK-58646][PS][FOLLOWUP] Fix NumPy reciprocal parity for decimal,
boolean, and narrower integer columns
### What changes were proposed in this pull request?
This is a follow-up of #57856 (SPARK-58646), which replaced the scalar
pandas UDF fallback of `np.reciprocal` with native Spark expressions for
non-floating-point inputs.
That change routed every non-`float`/`double` dtype through a single
integer branch that hard-codes the int64 minimum as the divide-by-zero sentinel
and casts the quotient through `long`. This does not match the previous pandas
UDF (`np.reciprocal` applied to the pandas `Series`) for decimals, booleans,
and narrower integers:
| dtype | input | previous UDF (`np.reciprocal` -> Double) | merged native
(#57856) |
|---|---|---|---|
| int64 (bigint) | `0` | `-9.2e18` (int64 min) | `-9.2e18` (unchanged) |
| int32 (int) | `0` | `-2147483648` (int32 min) | `-9.2e18` |
| int8/int16 (tinyint/smallint) | `0` | `0` (numpy `1 // 0` does not
overflow on narrow widths) | `-9.2e18` |
| boolean | `False` | `0.0` (numpy promotes bool to int8: `True -> 1`,
`False -> 0`) | `-9.2e18` |
| decimal | `2.5` | `0.4` (numpy takes a true floating reciprocal) | `0.0`
(truncated to long) |
This PR restores parity:
- Decimal inputs now flow through the floating-point reciprocal branch
(`typeof` starts with `decimal`), since numpy computes a true reciprocal for
them. A decimal `0` (which the old UDF could not handle --
`np.reciprocal(Decimal('0'))` raises `DivisionByZero`) now maps to `inf`,
consistent with the floating-point branch.
- The integer/boolean branch now picks the divide-by-zero sentinel by
column width -- int32 minimum for `int`, int64 minimum for `bigint`, and `0`
for the narrower widths (`tinyint`, `smallint`, and `boolean` promoted to int8)
-- and casts through `long` so boolean and narrower integers can take part in
the division.
int64 columns, the only case exercised by the original PR, are unchanged.
### Why are the changes needed?
The merged native expression regressed the observable pandas-on-Spark
behavior for decimal, boolean, and narrower-integer columns relative to the
pandas UDF it replaced. This restores parity so that `np.reciprocal` produces
the same results as before across all supported dtypes.
### Does this PR introduce _any_ user-facing change?
No. #57856 is unreleased (master only), so this only fixes an unreleased
regression before it ships; there is no change relative to any released Spark
version.
### How was this patch tested?
Added `test_np_reciprocal_non_default_dtypes` in
`python/pyspark/pandas/tests/test_numpy_compat.py`, inherited by the Spark
Connect parity suite, asserting `np.reciprocal(psser)` equals
`np.reciprocal(pdf)` for int8/int16/int32, boolean, and decimal columns
(covering positive, negative, and the per-width zero-overflow sentinel).
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #58218 from Yicong-Huang/reciprocal-nonfloat-fix.
Authored-by: Yicong Huang <[email protected]>
Signed-off-by: Yicong-Huang <[email protected]>
---
python/pyspark/pandas/numpy_compat.py | 30 ++++++++++++++++--------
python/pyspark/pandas/tests/test_numpy_compat.py | 26 ++++++++++++++++++++
2 files changed, 46 insertions(+), 10 deletions(-)
diff --git a/python/pyspark/pandas/numpy_compat.py
b/python/pyspark/pandas/numpy_compat.py
index 33ec402a2012..05dbc7c1c7b4 100644
--- a/python/pyspark/pandas/numpy_compat.py
+++ b/python/pyspark/pandas/numpy_compat.py
@@ -70,21 +70,31 @@ unary_np_spark_mappings = {
"rad2deg": F.degrees,
"radians": F.radians,
"reciprocal": lambda c: F.when(
- F.typeof(c).isin("float", "double"),
+ # Floating-point and decimal inputs take a true reciprocal; numpy
+ # applies it element-wise to Decimal objects as well.
+ F.typeof(c).isin("float", "double") |
F.typeof(c).startswith("decimal"),
F.when(c.isNull(), c.cast("double"))
.when(
- c == 0,
+ # Cast to double so the zero check also analyzes for the integer
and
+ # boolean columns that fall through to the otherwise branch (Spark
+ # type-checks every branch of the CASE, not just the taken one).
+ c.cast("double") == 0,
F.when(c.cast("string") == "-0.0",
F.lit(float("-inf"))).otherwise(F.lit(float("inf"))),
)
- .otherwise(F.lit(1.0) / c),
+ .otherwise(F.lit(1.0) / c.cast("double")),
).otherwise(
- # Integer input: numpy does integer division (truncated toward zero),
- # so casting the float quotient to long reproduces 1 -> 1, -1 -> -1,
- # and every other magnitude -> 0. Dividing by 0 overflows to the int64
- # minimum, matching numpy's behavior on integer arrays.
- F.when(c == 0, F.lit(float(np.iinfo(np.int64).min))).otherwise(
- (F.lit(1) / c).cast("long").cast("double")
- )
+ # Integer and boolean inputs: numpy does integer division (truncated
+ # toward zero), so only +/-1 survive and every other magnitude -> 0.
+ # Dividing by 0 overflows to the width-specific integer minimum for int
+ # (int32) and bigint (int64), while narrower widths (tinyint, smallint,
+ # and boolean promoted to int8) return 0. Cast through long so boolean
+ # and narrower integers can take part in the division.
+ F.when(
+ c.cast("long") == 0,
+ F.when(F.typeof(c) == "int", F.lit(float(np.iinfo(np.int32).min)))
+ .when(F.typeof(c) == "bigint",
F.lit(float(np.iinfo(np.int64).min)))
+ .otherwise(F.lit(0.0)),
+ ).otherwise((F.lit(1) / c.cast("long")).cast("long").cast("double"))
),
"rint": lambda c: F.rint(c.cast("double")),
"sign": F.signum,
diff --git a/python/pyspark/pandas/tests/test_numpy_compat.py
b/python/pyspark/pandas/tests/test_numpy_compat.py
index a3f92ef72058..b211d5d91aaa 100644
--- a/python/pyspark/pandas/tests/test_numpy_compat.py
+++ b/python/pyspark/pandas/tests/test_numpy_compat.py
@@ -17,6 +17,7 @@
import platform
import unittest
+from decimal import Decimal
import numpy as np
import pandas as pd
@@ -180,6 +181,31 @@ class NumPyCompatTestsMixin:
self.assert_eq(np.reciprocal(psdf.a), np.reciprocal(pdf.a),
almost=True)
+ @_skip_if_numpy_differs
+ def test_np_reciprocal_non_default_dtypes(self):
+ # The non-floating reciprocal branch also serves narrower integers,
+ # booleans, and decimals. numpy divides integers (and booleans, as
+ # int8) toward zero, so 0 overflows to the width-specific minimum
+ # (0 for int8/int16, int32 min for int32), while decimals take a true
+ # floating reciprocal. Lock in parity with pandas for each.
+ for dtype in ("int8", "int16", "int32"):
+ with self.subTest(dtype=dtype):
+ pdf = pd.DataFrame({"a": np.array([-2, -1, 0, 1, 2],
dtype=dtype)})
+ psdf = ps.from_pandas(pdf)
+
+ self.assert_eq(np.reciprocal(psdf.a), np.reciprocal(pdf.a),
almost=True)
+
+ # Boolean: numpy promotes to int8 (True -> 1, False -> 0).
+ pdf = pd.DataFrame({"a": [True, False, True]})
+ psdf = ps.from_pandas(pdf)
+ self.assert_eq(np.reciprocal(psdf.a), np.reciprocal(pdf.a),
almost=True)
+
+ # Decimal: numpy takes a floating reciprocal. 0 is excluded because
+ # numpy raises DivisionByZero on Decimal('0').
+ pdf = pd.DataFrame({"a": [Decimal("2.5"), Decimal("-4"),
Decimal("0.5")]})
+ psdf = ps.from_pandas(pdf)
+ self.assert_eq(np.reciprocal(psdf.a), np.reciprocal(pdf.a),
almost=True)
+
def test_np_bitwise_shift_functions(self):
pdf = pd.DataFrame(
{
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
