This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new b72d7412ee89 [SPARK-58445][PS] Fix NumPy isfinite and isinf mappings
b72d7412ee89 is described below
commit b72d7412ee8947ecda8671d0bba68b8d2cbfe92f
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Fri Jul 31 08:59:28 2026 +0800
[SPARK-58445][PS] Fix NumPy isfinite and isinf mappings
### What changes were proposed in this pull request?
Correct the pandas-on-Spark mappings for NumPy `isfinite` and `isinf`. Both
predicates now check positive and negative infinity, and `isfinite` also
excludes `NaN`. Add pandas-on-Spark parity coverage for finite values, both
infinities, signed zero, and `NaN`.
### Why are the changes needed?
This is a longstanding pandas-on-Spark bug that has existed since
pandas-on-Spark was first introduced. The existing mappings recognize only
positive infinity. As a result, `np.isfinite` incorrectly returned true for
negative infinity and `NaN`, while `np.isinf` incorrectly returned false for
negative infinity.
### Does this PR introduce _any_ user-facing change?
Yes. `np.isfinite` and `np.isinf` on pandas-on-Spark objects now return
NumPy-compatible results for negative infinity and `NaN`.
### How was this patch tested?
- Added pandas-on-Spark parity coverage for finite values, infinities,
signed zero, and `NaN`.
- Ran `build/sbt -java-home /usr/lib/jvm/java-17-openjdk-amd64 -Phive
package`.
- Ran `JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 SPARK_TESTING=1
SPARK_PREPEND_CLASSES=1 PYSPARK_PYTHON=.venv/bin/python
PYSPARK_DRIVER_PYTHON=.venv/bin/python python/run-tests --testnames
pyspark.pandas.tests.test_numpy_compat`.
- Ran `git diff --check`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
Closes #57586 from zhengruifeng/pandas-numpy-finite-ufuncs.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
(cherry picked from commit 24d7b0c31a2576bd2bb57aaf97c0c20dfbb98055)
Signed-off-by: Ruifeng Zheng <[email protected]>
---
python/pyspark/pandas/numpy_compat.py | 6 ++++--
python/pyspark/pandas/tests/test_numpy_compat.py | 2 ++
2 files changed, 6 insertions(+), 2 deletions(-)
diff --git a/python/pyspark/pandas/numpy_compat.py
b/python/pyspark/pandas/numpy_compat.py
index 09194cbee5e0..febba91883bd 100644
--- a/python/pyspark/pandas/numpy_compat.py
+++ b/python/pyspark/pandas/numpy_compat.py
@@ -50,8 +50,10 @@ unary_np_spark_mappings = {
"frexp": lambda _: NotImplemented, # 'frexp' output lengths become
different
# and it cannot be supported via pandas UDF.
"invert": F.bitwise_not,
- "isfinite": lambda c: c != float("inf"),
- "isinf": lambda c: c == float("inf"),
+ "isfinite": lambda c: F.coalesce(
+ ~(F.isnan(c) | (c == float("inf")) | (c == float("-inf"))),
F.lit(False)
+ ),
+ "isinf": lambda c: F.coalesce((c == float("inf")) | (c == float("-inf")),
F.lit(False)),
"isnan": F.isnan,
"isnat": lambda c: NotImplemented, # pandas-on-Spark and PySpark does not
have Nat concept.
"log": F.log,
diff --git a/python/pyspark/pandas/tests/test_numpy_compat.py
b/python/pyspark/pandas/tests/test_numpy_compat.py
index a3234ed4169d..432283cb5fe8 100644
--- a/python/pyspark/pandas/tests/test_numpy_compat.py
+++ b/python/pyspark/pandas/tests/test_numpy_compat.py
@@ -99,6 +99,8 @@ class NumPyCompatTestsMixin:
(np.fabs, [np.iinfo(np.int64).min, -2, 0, 2]),
(np.fabs, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf, np.nan]),
(np.invert, [np.iinfo(np.int64).min, -2, -1, 0, 1, 2,
np.iinfo(np.int64).max]),
+ (np.isfinite, [-np.inf, -64.0, -0.0, 0.0, 64.0, np.inf, np.nan]),
+ (np.isinf, [-np.inf, -64.0, -0.0, 0.0, 64.0, np.inf, np.nan]),
(np.negative, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf,
np.nan]),
(np.positive, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf,
np.nan]),
(np.rad2deg, [-np.inf, -64.0, -np.pi, 0.0, np.pi, 64.0, np.inf,
np.nan]),
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]