This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 0e22d358ad4a [SPARK-58553][PS][FOLLOWUP] Preserve signed zero in NumPy
fmax and fmin
0e22d358ad4a is described below
commit 0e22d358ad4a6bf31dab9df651990e9981d254a9
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Mon Aug 10 12:12:20 2026 +0800
[SPARK-58553][PS][FOLLOWUP] Preserve signed zero in NumPy fmax and fmin
### What changes were proposed in this pull request?
This follow-up fixes the native `np.fmax` and `np.fmin` mappings to
preserve the first operand when the operands compare equal. This preserves
NumPy signed-zero behavior while retaining the existing NaN handling and native
Spark expressions.
### Why are the changes needed?
Spark `greatest` and `least` order `-0.0` and `+0.0`, whereas NumPy retains
the first operand for an equal-value tie. Consequently, the prior mapping
returned the wrong signed zero for `fmax(-0.0, 0.0)` and `fmin(0.0, -0.0)`.
### Does this PR introduce _any_ user-facing change?
Yes. It corrects the sign of zero for the affected `np.fmax` and `np.fmin`
inputs.
### How was this patch tested?
Added signbit-based regression coverage for both signed-zero operand orders.
- `python/run-tests --testnames
pyspark.pandas.tests.test_numpy_compat.NumPyCompatTests.test_np_fmax_fmin`
- `python/run-tests --testnames
pyspark.pandas.tests.connect.test_parity_numpy_compat.NumPyCompatParityTests.test_np_fmax_fmin`
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex GPT-5
Closes #57819 from zhengruifeng/pandas-native-fmax-fmin-followup-dev2.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
(cherry picked from commit 32a65fbe54913282c2f3579e9e443dd3d63f7fd4)
Signed-off-by: Ruifeng Zheng <[email protected]>
---
python/pyspark/pandas/numpy_compat.py | 3 ++-
python/pyspark/pandas/tests/test_numpy_compat.py | 7 +++++++
2 files changed, 9 insertions(+), 1 deletion(-)
diff --git a/python/pyspark/pandas/numpy_compat.py
b/python/pyspark/pandas/numpy_compat.py
index 33538ef9de87..3df41058f91b 100644
--- a/python/pyspark/pandas/numpy_compat.py
+++ b/python/pyspark/pandas/numpy_compat.py
@@ -132,9 +132,10 @@ binary_np_spark_mappings = {
),
"fmax": lambda c1, c2: F.when(F.isnan(c1.cast("double")), c2)
.when(F.isnan(c2.cast("double")), c1)
+ .when(c1 == c2, c1)
.otherwise(F.greatest(c1, c2))
.cast("double"),
- "fmin": lambda c1, c2: F.least(c1, c2).cast("double"),
+ "fmin": lambda c1, c2: F.when(c1 == c2, c1).otherwise(F.least(c1,
c2)).cast("double"),
"fmod": _fmod_func,
"gcd": pandas_udf(lambda s1, s2: np.gcd(s1, s2), DoubleType()), # type:
ignore[call-overload]
"heaviside": lambda c1, c2: F.when(
diff --git a/python/pyspark/pandas/tests/test_numpy_compat.py
b/python/pyspark/pandas/tests/test_numpy_compat.py
index d95474f7fc07..ab4537498016 100644
--- a/python/pyspark/pandas/tests/test_numpy_compat.py
+++ b/python/pyspark/pandas/tests/test_numpy_compat.py
@@ -255,12 +255,19 @@ class NumPyCompatTestsMixin:
"x2": [2.0, np.nan, np.nan, np.inf, -np.inf, 0.0, -0.0,
np.inf, -np.inf],
}
),
+ pd.DataFrame({"x1": [-0.0, 0.0], "x2": [0.0, -0.0]}),
):
psdf = ps.from_pandas(pdf)
for np_func in (np.fmax, np.fmin):
result = np_func(psdf.x1, psdf.x2)
expected = np_func(pdf.x1, pdf.x2)
self.assert_eq(result, expected, almost=True)
+ # NumPy's vectorized implementation may select either zero
operand, whereas
+ # its scalar implementation consistently selects the first one.
+ expected_signbit = pd.Series(
+ [np.signbit(np_func(x1, x2)) for x1, x2 in zip(pdf.x1,
pdf.x2)]
+ )
+ self.assert_eq(np.signbit(result.to_pandas()),
expected_signbit)
def test_np_heaviside(self):
for pdf in (
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
