This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 29913a0e84ba [SPARK-58718][PS] Fix signbit for -0.0 and nullable dtypes
29913a0e84ba is described below
commit 29913a0e84badab17fd9121786a32731ac0bd77f
Author: Spenser Sun <[email protected]>
AuthorDate: Wed Aug 12 10:26:37 2026 +0800
[SPARK-58718][PS] Fix signbit for -0.0 and nullable dtypes
### What changes were proposed in this pull request?
`np.signbit` in the pandas API on Spark is mapped in `numpy_compat.py` to
`F.when(c < 0, True).otherwise(False)`. This PR replaces that mapping so it
matches NumPy on two points:
1. **Signed zero.** `np.signbit` returns the IEEE-754 sign bit, so
`np.signbit(-0.0)` is `True`. The old `c < 0` test is `False` for `-0.0`
(numerically `-0.0 == 0.0`). The new mapping also treats `-0.0` as signed via a
string cast (`c.cast("string") == "-0.0"`) — the same idiom already used by the
`reciprocal` and `copysign` mappings.
2. **Null propagation for nullable dtypes.** A genuine `<NA>` from a
nullable extension dtype (e.g. `Int64`) should propagate to a null result; the
old mapping returned `False` for it.
The subtle part is that `ps.from_pandas` collapses a default-dtype `NaN`
and a nullable-dtype `<NA>` onto the same Spark value (`isNull = true`)
whenever the column is `double`, so at the expression level they are
indistinguishable. The mapping therefore keys on `typeof` to separate the cases
it can. The four scenarios:
| pandas input | Spark value after `from_pandas` | `np.signbit` reference |
this PR |
|---|---|---|---|
| default `int` with a missing value (upcasts to float, `None` -> `NaN`) |
`double` null | `False` | `False` |
| default `float64` `NaN` | `double` null | `False` | `False` |
| nullable `Int64` `<NA>` | `bigint` null | propagate -> null | propagate
-> null |
| nullable `Float64` `<NA>` | `double` null | propagate -> null | `False`
(known gap) |
The last row is a known limitation: a nullable `Float64` `<NA>` becomes a
`double` null, byte-identical to a default-float `NaN`, so it cannot be
distinguished after `from_pandas`. Since the common default-float path wants
`False`, `double` nulls map to `False` and the exotic nullable-`Float64` `<NA>`
is sacrificed.
Resulting mapping:
```python
"signbit": lambda c: F.when(
c.isNull() & ~F.typeof(c).isin("float", "double"),
F.lit(None).cast("boolean"),
)
.when((c < 0) | (c.cast("string") == "-0.0"), True)
.otherwise(False),
```
This follows the same `typeof`-keyed approach and the same trade-off as the
recent `copysign` fix (SPARK-58630, #57836 by Yicong-Huang). The design is
still open for discussion — in particular the choice to sacrifice the
nullable-`Float64` `<NA>` case rather than always propagating or always
returning `False`.
### Why are the changes needed?
`np.signbit` on a pandas-on-Spark object disagrees with NumPy/pandas for
`-0.0` (returns `False` instead of `True`) and does not propagate missing
values for nullable dtypes. The mapping should match the NumPy reference that
the compat tests assert against. The `-0.0` divergence went unnoticed because
the only existing coverage is a sweep over random integers, which never
produces `-0.0` or a null.
### Does this PR introduce _any_ user-facing change?
Yes. `np.signbit` on a pandas-on-Spark Series/DataFrame now matches NumPy.
Before:
```python
>>> import numpy as np, pyspark.pandas as ps
>>> np.signbit(ps.Series([-0.0, 0.0, -1.0, 1.0]))
0 False
1 False
2 True
3 False
```
After:
```python
>>> np.signbit(ps.Series([-0.0, 0.0, -1.0, 1.0]))
0 True
1 False
2 True
3 False
```
A genuine `<NA>` in a nullable dtype (e.g. `Int64`) now propagates as null
instead of returning `False`.
### How was this patch tested?
Added `test_np_signbit` to `NumPyCompatTestsMixin` (inherited by the Spark
Connect parity suite), covering `-0.0`, `+0.0`, `±inf`, `NaN`, a default `int`
column with a missing value, and a nullable `Int64` column with `<NA>`. Ran the
full `NumPyCompatTests` suite locally (classic mode): all pass, including the
generic compat sweep that also exercises `signbit`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)
Closes #57937 from Spenserrrr/numpy-signbit-signed-zero.
Authored-by: Spenser Sun <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
(cherry picked from commit 0ef1c73e7a755ec00b3b685ec424cabb8799af5b)
Signed-off-by: Ruifeng Zheng <[email protected]>
---
python/pyspark/pandas/numpy_compat.py | 12 +++++++++++-
python/pyspark/pandas/tests/test_numpy_compat.py | 15 +++++++++++++++
2 files changed, 26 insertions(+), 1 deletion(-)
diff --git a/python/pyspark/pandas/numpy_compat.py
b/python/pyspark/pandas/numpy_compat.py
index dea76a4cd9ec..f361e02712f9 100644
--- a/python/pyspark/pandas/numpy_compat.py
+++ b/python/pyspark/pandas/numpy_compat.py
@@ -86,7 +86,17 @@ unary_np_spark_mappings = {
),
"rint": lambda c: F.rint(c.cast("double")),
"sign": F.signum,
- "signbit": lambda c: F.when(c < 0, True).otherwise(False),
+ "signbit": lambda c: F.when(
+ # A genuine <NA> from a nullable dtype (e.g. Int64) arrives as a
non-double
+ # null and must propagate. A NaN from a default (numpy-backed) dtype
arrives
+ # as a double null and must map to False (np.signbit(nan) is False);
it falls
+ # through to otherwise(False) below. A nullable Float64 <NA> is also a
double
+ # null, indistinguishable from a NaN after from_pandas, so it cannot
propagate.
+ c.isNull() & ~F.typeof(c).isin("float", "double"),
+ F.lit(None).cast("boolean"),
+ )
+ .when((c < 0) | (c.cast("string") == "-0.0"), True)
+ .otherwise(False),
"sin": F.sin,
"sinh": F.sinh,
"spacing": pandas_udf(lambda s: np.spacing(s), DoubleType()), # type:
ignore[call-overload]
diff --git a/python/pyspark/pandas/tests/test_numpy_compat.py
b/python/pyspark/pandas/tests/test_numpy_compat.py
index 0e5aaf47df23..79c1fb1da7e6 100644
--- a/python/pyspark/pandas/tests/test_numpy_compat.py
+++ b/python/pyspark/pandas/tests/test_numpy_compat.py
@@ -368,6 +368,21 @@ class NumPyCompatTestsMixin:
np.heaviside(psdf.x1, psdf.x2), np.heaviside(pdf.x1, pdf.x2),
almost=True
)
+ def test_np_signbit(self):
+ # np.signbit returns the IEEE-754 sign bit, which differs from (x < 0)
only
+ # at -0.0: the sign bit is set even though -0.0 is not less than zero.
A
+ # missing value in a default (numpy-backed) dtype arrives as a NaN and
maps
+ # to False (np.signbit(nan) is False), whereas a genuine <NA> in a
nullable
+ # dtype (e.g. Int64) propagates. A nullable Float64 <NA> is
indistinguishable
+ # from a NaN after from_pandas, so it is deliberately not covered here.
+ for pdf in (
+ pd.DataFrame({"a": [-0.0, 0.0, -1.0, 1.0, -np.inf, np.inf,
np.nan]}),
+ pd.DataFrame({"a": [1, -2, None]}),
+ pd.DataFrame({"a": pd.array([1, -2, None], dtype="Int64")}),
+ ):
+ psdf = ps.from_pandas(pdf)
+ self.assert_eq(np.signbit(psdf.a), np.signbit(pdf.a))
+
def test_np_spark_compat_series(self):
from pyspark.pandas.numpy_compat import unary_np_spark_mappings,
binary_np_spark_mappings
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
