This is an automated email from the ASF dual-hosted git repository.

zhengruifeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 67a9b0dc89af [SPARK-58465][PS] Use native expressions for NumPy 
reciprocal of floating-point columns
67a9b0dc89af is described below

commit 67a9b0dc89af0be7e391860e5bb0aae12248b30a
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Tue Aug 4 11:12:34 2026 +0800

    [SPARK-58465][PS] Use native expressions for NumPy reciprocal of 
floating-point columns
    
    ### What changes were proposed in this pull request?
    
    Use native Spark SQL expressions for NumPy reciprocal on pandas-on-Spark 
columns with Spark `float` or `double` types. The expressions preserve nulls, 
NaN, infinity, and signed zero. Inputs with other types continue to use the 
existing pandas UDF.
    
    ### Why are the changes needed?
    
    Native expressions avoid the Python worker boundary for floating-point 
reciprocal while retaining the established behavior for input types with 
NumPy-specific semantics.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. The behavior and result type remain unchanged.
    
    ### How was this patch tested?
    
    - Added reciprocal coverage for negative zero in 
`NumPyCompatTests.test_np_math_functions`.
    - Passed `NumPyCompatTests.test_np_math_functions` and 
`NumPyCompatTests.test_np_spark_compat_series`.
    - Passed Ruff format and lint checks for 
`python/pyspark/pandas/numpy_compat.py`.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Codex (GPT-5)
    
    Closes #57668 from zhengruifeng/pandas-native-reciprocal-dev2.
    
    Authored-by: Ruifeng Zheng <[email protected]>
    Signed-off-by: Ruifeng Zheng <[email protected]>
---
 python/pyspark/pandas/numpy_compat.py            | 12 ++++++++++--
 python/pyspark/pandas/tests/test_numpy_compat.py |  4 ++++
 2 files changed, 14 insertions(+), 2 deletions(-)

diff --git a/python/pyspark/pandas/numpy_compat.py 
b/python/pyspark/pandas/numpy_compat.py
index 1653121d3124..2a39fd65d933 100644
--- a/python/pyspark/pandas/numpy_compat.py
+++ b/python/pyspark/pandas/numpy_compat.py
@@ -66,8 +66,16 @@ unary_np_spark_mappings = {
     "positive": F.positive,
     "rad2deg": F.degrees,
     "radians": F.radians,
-    "reciprocal": pandas_udf(  # type: ignore[call-overload]
-        lambda s: np.reciprocal(s), DoubleType()
+    "reciprocal": lambda c: F.when(
+        F.typeof(c).isin("float", "double"),
+        F.when(c.isNull(), c.cast("double"))
+        .when(
+            c == 0,
+            F.when(c.cast("string") == "-0.0", 
F.lit(float("-inf"))).otherwise(F.lit(float("inf"))),
+        )
+        .otherwise(F.lit(1.0) / c),
+    ).otherwise(
+        pandas_udf(np.reciprocal, DoubleType())(c)  # type: 
ignore[call-overload]
     ),
     "rint": lambda c: F.rint(c.cast("double")),
     "sign": F.signum,
diff --git a/python/pyspark/pandas/tests/test_numpy_compat.py 
b/python/pyspark/pandas/tests/test_numpy_compat.py
index 2530d6b04a80..719e6e0114b2 100644
--- a/python/pyspark/pandas/tests/test_numpy_compat.py
+++ b/python/pyspark/pandas/tests/test_numpy_compat.py
@@ -105,6 +105,10 @@ class NumPyCompatTestsMixin:
             (np.positive, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf, 
np.nan]),
             (np.rad2deg, [-np.inf, -64.0, -np.pi, 0.0, np.pi, 64.0, np.inf, 
np.nan]),
             (np.rint, [-np.inf, -2.5, -1.5, -0.5, -0.0, 0.0, 0.5, 1.5, 2.5, 
np.inf, np.nan]),
+            (
+                np.reciprocal,
+                [-np.inf, -64.0, -2.0, -1.0, -0.0, 0.0, 1.0, 2.0, 64.0, 
np.inf, np.nan],
+            ),
             (np.sign, [-np.inf, -64.0, -2.0, -0.0, 0.0, 2.0, 64.0, np.inf, 
np.nan]),
             (np.sinh, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf, np.nan]),
             (np.square, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf, 
np.nan]),


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to