This is an automated email from the ASF dual-hosted git repository.

zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new b72d7412ee89 [SPARK-58445][PS] Fix NumPy isfinite and isinf mappings
b72d7412ee89 is described below

commit b72d7412ee8947ecda8671d0bba68b8d2cbfe92f
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Fri Jul 31 08:59:28 2026 +0800

    [SPARK-58445][PS] Fix NumPy isfinite and isinf mappings
    
    ### What changes were proposed in this pull request?
    
    Correct the pandas-on-Spark mappings for NumPy `isfinite` and `isinf`. Both 
predicates now check positive and negative infinity, and `isfinite` also 
excludes `NaN`. Add pandas-on-Spark parity coverage for finite values, both 
infinities, signed zero, and `NaN`.
    
    ### Why are the changes needed?
    
    This is a longstanding pandas-on-Spark bug that has existed since 
pandas-on-Spark was first introduced. The existing mappings recognize only 
positive infinity. As a result, `np.isfinite` incorrectly returned true for 
negative infinity and `NaN`, while `np.isinf` incorrectly returned false for 
negative infinity.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `np.isfinite` and `np.isinf` on pandas-on-Spark objects now return 
NumPy-compatible results for negative infinity and `NaN`.
    
    ### How was this patch tested?
    
    - Added pandas-on-Spark parity coverage for finite values, infinities, 
signed zero, and `NaN`.
    - Ran `build/sbt -java-home /usr/lib/jvm/java-17-openjdk-amd64 -Phive 
package`.
    - Ran `JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 SPARK_TESTING=1 
SPARK_PREPEND_CLASSES=1 PYSPARK_PYTHON=.venv/bin/python 
PYSPARK_DRIVER_PYTHON=.venv/bin/python python/run-tests --testnames 
pyspark.pandas.tests.test_numpy_compat`.
    - Ran `git diff --check`.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Codex (GPT-5)
    
    Closes #57586 from zhengruifeng/pandas-numpy-finite-ufuncs.
    
    Authored-by: Ruifeng Zheng <[email protected]>
    Signed-off-by: Ruifeng Zheng <[email protected]>
    (cherry picked from commit 24d7b0c31a2576bd2bb57aaf97c0c20dfbb98055)
    Signed-off-by: Ruifeng Zheng <[email protected]>
---
 python/pyspark/pandas/numpy_compat.py            | 6 ++++--
 python/pyspark/pandas/tests/test_numpy_compat.py | 2 ++
 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/python/pyspark/pandas/numpy_compat.py 
b/python/pyspark/pandas/numpy_compat.py
index 09194cbee5e0..febba91883bd 100644
--- a/python/pyspark/pandas/numpy_compat.py
+++ b/python/pyspark/pandas/numpy_compat.py
@@ -50,8 +50,10 @@ unary_np_spark_mappings = {
     "frexp": lambda _: NotImplemented,  # 'frexp' output lengths become 
different
     # and it cannot be supported via pandas UDF.
     "invert": F.bitwise_not,
-    "isfinite": lambda c: c != float("inf"),
-    "isinf": lambda c: c == float("inf"),
+    "isfinite": lambda c: F.coalesce(
+        ~(F.isnan(c) | (c == float("inf")) | (c == float("-inf"))), 
F.lit(False)
+    ),
+    "isinf": lambda c: F.coalesce((c == float("inf")) | (c == float("-inf")), 
F.lit(False)),
     "isnan": F.isnan,
     "isnat": lambda c: NotImplemented,  # pandas-on-Spark and PySpark does not 
have Nat concept.
     "log": F.log,
diff --git a/python/pyspark/pandas/tests/test_numpy_compat.py 
b/python/pyspark/pandas/tests/test_numpy_compat.py
index a3234ed4169d..432283cb5fe8 100644
--- a/python/pyspark/pandas/tests/test_numpy_compat.py
+++ b/python/pyspark/pandas/tests/test_numpy_compat.py
@@ -99,6 +99,8 @@ class NumPyCompatTestsMixin:
             (np.fabs, [np.iinfo(np.int64).min, -2, 0, 2]),
             (np.fabs, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf, np.nan]),
             (np.invert, [np.iinfo(np.int64).min, -2, -1, 0, 1, 2, 
np.iinfo(np.int64).max]),
+            (np.isfinite, [-np.inf, -64.0, -0.0, 0.0, 64.0, np.inf, np.nan]),
+            (np.isinf, [-np.inf, -64.0, -0.0, 0.0, 64.0, np.inf, np.nan]),
             (np.negative, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf, 
np.nan]),
             (np.positive, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf, 
np.nan]),
             (np.rad2deg, [-np.inf, -64.0, -np.pi, 0.0, np.pi, 64.0, np.inf, 
np.nan]),


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to