This is an automated email from the ASF dual-hosted git repository.
Yicong-Huang pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new a48c8bc59e37 [SPARK-58646][PS] Use native expressions for NumPy
reciprocal on integer columns
a48c8bc59e37 is described below
commit a48c8bc59e37eb0318a1a891195c9e7448dc14e6
Author: Yicong Huang <[email protected]>
AuthorDate: Tue Aug 11 00:02:23 2026 +0000
[SPARK-58646][PS] Use native expressions for NumPy reciprocal on integer
columns
### What changes were proposed in this pull request?
This PR replaces the Python `pandas_udf` fallback used by `np.reciprocal`
on integer columns in pandas-on-Spark with a native Spark SQL expression. The
float/double branch of the `reciprocal` mapping in
`python/pyspark/pandas/numpy_compat.py` was already native; only the integer
fallback still wrapped `np.reciprocal` in a scalar pandas UDF. The new
expression reproduces NumPy's integer semantics natively: integer division
truncated toward zero (`1 -> 1`, `-1 -> -1`, every other magnit [...]
### Why are the changes needed?
Evaluating a Python UDF per batch incurs serialization to and from the
Python worker, which is far more expensive than a native Catalyst expression
that runs entirely in the JVM. Removing the UDF for the integer path avoids
that round trip and keeps the behavior identical to the previous implementation.
### Does this PR introduce _any_ user-facing change?
No. The output values are unchanged for all integer inputs (verified
against `np.reciprocal` on the equivalent pandas Series for positive, negative,
zero, and int64 boundary values).
### How was this patch tested?
Added `test_np_reciprocal_integer` to
`python/pyspark/pandas/tests/test_numpy_compat.py` covering positive, negative,
and zero integer inputs (including the int64 min/max boundaries), asserting
parity with `np.reciprocal` on the pandas reference. The test is inherited by
the Spark Connect parity suite (`test_parity_numpy_compat.py`). Both the
classic and Connect suites pass.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #57856 from Yicong-Huang/reciprocal-int.
Authored-by: Yicong Huang <[email protected]>
Signed-off-by: Yicong-Huang <[email protected]>
(cherry picked from commit 2595e51f1eccb06b27cf697855bd9395eb9da17c)
Signed-off-by: Yicong-Huang <[email protected]>
---
python/pyspark/pandas/numpy_compat.py | 8 +++++++-
python/pyspark/pandas/tests/test_numpy_compat.py | 17 +++++++++++++++++
2 files changed, 24 insertions(+), 1 deletion(-)
diff --git a/python/pyspark/pandas/numpy_compat.py
b/python/pyspark/pandas/numpy_compat.py
index 3df41058f91b..d2e5fb01597c 100644
--- a/python/pyspark/pandas/numpy_compat.py
+++ b/python/pyspark/pandas/numpy_compat.py
@@ -76,7 +76,13 @@ unary_np_spark_mappings = {
)
.otherwise(F.lit(1.0) / c),
).otherwise(
- pandas_udf(np.reciprocal, DoubleType())(c) # type:
ignore[call-overload]
+ # Integer input: numpy does integer division (truncated toward zero),
+ # so casting the float quotient to long reproduces 1 -> 1, -1 -> -1,
+ # and every other magnitude -> 0. Dividing by 0 overflows to the int64
+ # minimum, matching numpy's behavior on integer arrays.
+ F.when(c == 0, F.lit(float(np.iinfo(np.int64).min))).otherwise(
+ (F.lit(1) / c).cast("long").cast("double")
+ )
),
"rint": lambda c: F.rint(c.cast("double")),
"sign": F.signum,
diff --git a/python/pyspark/pandas/tests/test_numpy_compat.py
b/python/pyspark/pandas/tests/test_numpy_compat.py
index ab4537498016..2573ff586c87 100644
--- a/python/pyspark/pandas/tests/test_numpy_compat.py
+++ b/python/pyspark/pandas/tests/test_numpy_compat.py
@@ -139,6 +139,23 @@ class NumPyCompatTestsMixin:
self.assert_eq(np_func(psdf.a), np_func(pdf.a), almost=True)
+ def test_np_reciprocal_integer(self):
+ # np.reciprocal on an integer column does integer division (truncated
+ # toward zero): 1 -> 1, -1 -> -1, and every other magnitude -> 0. The
+ # value 0 overflows to the int64 minimum. Cover positive, negative, and
+ # zero inputs to lock in parity with pandas.
+ for values in (
+ [1, 2, 3, 64, 100],
+ [-1, -2, -3, -64, -100],
+ [-2, -1, 0, 1, 2],
+ [np.iinfo(np.int64).min, -1, 1, np.iinfo(np.int64).max],
+ ):
+ with self.subTest(values=values):
+ pdf = pd.DataFrame({"a": values})
+ psdf = ps.from_pandas(pdf)
+
+ self.assert_eq(np.reciprocal(psdf.a), np.reciprocal(pdf.a),
almost=True)
+
def test_np_bitwise_shift_functions(self):
pdf = pd.DataFrame(
{
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
