This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 1762c910dfd5 [SPARK-58553][PS] Use native Spark functions for NumPy
fmax and fmin
1762c910dfd5 is described below
commit 1762c910dfd58a4b2d75447bdc5aacd4756d6d9c
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Wed Aug 5 09:53:29 2026 +0800
[SPARK-58553][PS] Use native Spark functions for NumPy fmax and fmin
### What changes were proposed in this pull request?
This PR replaces the pandas UDF implementations of `np.fmax` and `np.fmin`
in pandas API on Spark with native Spark expressions.
`fmax` explicitly selects the non-NaN operand before using `greatest`.
`fmin` uses `least`, whose NaN ordering matches NumPy `fmin`. Both results are
cast to `double`, matching the existing pandas UDF return type.
### Why are the changes needed?
Using native Spark expressions avoids pandas UDF and Arrow overhead while
preserving NumPy NaN-handling semantics.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Added `NumPyCompatTests.test_np_fmax_fmin`, covering integral inputs, NaNs,
infinities, and signed zero.
- `ruff check python/pyspark/pandas/numpy_compat.py
python/pyspark/pandas/tests/test_numpy_compat.py`
- `ruff format --check python/pyspark/pandas/numpy_compat.py
python/pyspark/pandas/tests/test_numpy_compat.py`
- `python -m unittest
pyspark.pandas.tests.test_numpy_compat.NumPyCompatTests.test_np_fmax_fmin`
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex GPT-5
Closes #57758 from zhengruifeng/pandas-native-fmax-fmin-dev2.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
---
python/pyspark/pandas/numpy_compat.py | 7 +++++--
python/pyspark/pandas/tests/test_numpy_compat.py | 16 ++++++++++++++++
2 files changed, 21 insertions(+), 2 deletions(-)
diff --git a/python/pyspark/pandas/numpy_compat.py
b/python/pyspark/pandas/numpy_compat.py
index f333594fd08d..f349b96411a2 100644
--- a/python/pyspark/pandas/numpy_compat.py
+++ b/python/pyspark/pandas/numpy_compat.py
@@ -111,8 +111,11 @@ binary_np_spark_mappings = {
"floor_divide": pandas_udf( # type: ignore[call-overload]
lambda s1, s2: np.floor_divide(s1, s2), DoubleType()
),
- "fmax": pandas_udf(lambda s1, s2: np.fmax(s1, s2), DoubleType()), # type:
ignore[call-overload]
- "fmin": pandas_udf(lambda s1, s2: np.fmin(s1, s2), DoubleType()), # type:
ignore[call-overload]
+ "fmax": lambda c1, c2: F.when(F.isnan(c1.cast("double")), c2)
+ .when(F.isnan(c2.cast("double")), c1)
+ .otherwise(F.greatest(c1, c2))
+ .cast("double"),
+ "fmin": lambda c1, c2: F.least(c1, c2).cast("double"),
"fmod": pandas_udf(lambda s1, s2: np.fmod(s1, s2), DoubleType()), # type:
ignore[call-overload]
"gcd": pandas_udf(lambda s1, s2: np.gcd(s1, s2), DoubleType()), # type:
ignore[call-overload]
"heaviside": pandas_udf( # type: ignore[call-overload]
diff --git a/python/pyspark/pandas/tests/test_numpy_compat.py
b/python/pyspark/pandas/tests/test_numpy_compat.py
index 53bdcc58e36c..ff74b792434c 100644
--- a/python/pyspark/pandas/tests/test_numpy_compat.py
+++ b/python/pyspark/pandas/tests/test_numpy_compat.py
@@ -171,6 +171,22 @@ class NumPyCompatTestsMixin:
almost=True,
)
+ def test_np_fmax_fmin(self):
+ for pdf in (
+ pd.DataFrame({"x1": [-2, -1, 0, 1, 2], "x2": [2, 1, 0, -1, -2]}),
+ pd.DataFrame(
+ {
+ "x1": [np.nan, 2.0, np.nan, -np.inf, -2.0, -0.0, 0.0, 2.0,
np.inf],
+ "x2": [2.0, np.nan, np.nan, np.inf, -np.inf, 0.0, -0.0,
np.inf, -np.inf],
+ }
+ ),
+ ):
+ psdf = ps.from_pandas(pdf)
+ for np_func in (np.fmax, np.fmin):
+ result = np_func(psdf.x1, psdf.x2)
+ expected = np_func(pdf.x1, pdf.x2)
+ self.assert_eq(result, expected, almost=True)
+
def test_np_spark_compat_series(self):
from pyspark.pandas.numpy_compat import unary_np_spark_mappings,
binary_np_spark_mappings
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
