Spenserrrr opened a new pull request, #57937:
URL: https://github.com/apache/spark/pull/57937
### What changes were proposed in this pull request?
`np.signbit` in the pandas API on Spark is mapped in `numpy_compat.py` to
`F.when(c < 0, True).otherwise(False)`. This PR replaces that mapping so it
matches NumPy on two points:
1. **Signed zero.** `np.signbit` returns the IEEE-754 sign bit, so
`np.signbit(-0.0)` is `True`. The old `c < 0` test is `False` for `-0.0`
(numerically `-0.0 == 0.0`). The new mapping also treats `-0.0` as signed via a
string cast (`c.cast("string") == "-0.0"`) — the same idiom already used by the
`reciprocal` and `copysign` mappings.
2. **Null propagation for nullable dtypes.** A genuine `<NA>` from a
nullable extension dtype (e.g. `Int64`) should propagate to a null result; the
old mapping returned `False` for it.
The subtle part is that `ps.from_pandas` collapses a default-dtype `NaN` and
a nullable-dtype `<NA>` onto the same Spark value (`isNull = true`) whenever
the column is `double`, so at the expression level they are indistinguishable.
The mapping therefore keys on `typeof` to separate the cases it can. The four
scenarios:
| pandas input | Spark value after `from_pandas` | `np.signbit` reference |
this PR |
|---|---|---|---|
| default `int` with a missing value (upcasts to float, `None` -> `NaN`) |
`double` null | `False` | `False` |
| default `float64` `NaN` | `double` null | `False` | `False` |
| nullable `Int64` `<NA>` | `bigint` null | propagate -> null | propagate ->
null |
| nullable `Float64` `<NA>` | `double` null | propagate -> null | `False`
(known gap) |
The last row is a known limitation: a nullable `Float64` `<NA>` becomes a
`double` null, byte-identical to a default-float `NaN`, so it cannot be
distinguished after `from_pandas`. Since the common default-float path wants
`False`, `double` nulls map to `False` and the exotic nullable-`Float64` `<NA>`
is sacrificed.
Resulting mapping:
```python
"signbit": lambda c: F.when(
c.isNull() & ~F.typeof(c).isin("float", "double"),
F.lit(None).cast("boolean"),
)
.when((c < 0) | (c.cast("string") == "-0.0"), True)
.otherwise(False),
```
This follows the same `typeof`-keyed approach and the same trade-off as the
recent `copysign` fix (SPARK-58630, #57836 by @Yicong-Huang). The design is
still open for discussion — in particular the choice to sacrifice the
nullable-`Float64` `<NA>` case rather than always propagating or always
returning `False`.
### Why are the changes needed?
`np.signbit` on a pandas-on-Spark object disagrees with NumPy/pandas for
`-0.0` (returns `False` instead of `True`) and does not propagate missing
values for nullable dtypes. The mapping should match the NumPy reference that
the compat tests assert against. The `-0.0` divergence went unnoticed because
the only existing coverage is a sweep over random integers, which never
produces `-0.0` or a null.
### Does this PR introduce _any_ user-facing change?
Yes. `np.signbit` on a pandas-on-Spark Series/DataFrame now matches NumPy.
Before:
```python
>>> import numpy as np, pyspark.pandas as ps
>>> np.signbit(ps.Series([-0.0, 0.0, -1.0, 1.0]))
0 False
1 False
2 True
3 False
```
After:
```python
>>> np.signbit(ps.Series([-0.0, 0.0, -1.0, 1.0]))
0 True
1 False
2 True
3 False
```
A genuine `<NA>` in a nullable dtype (e.g. `Int64`) now propagates as null
instead of returning `False`.
### How was this patch tested?
Added `test_np_signbit` to `NumPyCompatTestsMixin` (inherited by the Spark
Connect parity suite), covering `-0.0`, `+0.0`, `±inf`, `NaN`, a default `int`
column with a missing value, and a nullable `Int64` column with `<NA>`. Ran the
full `NumPyCompatTests` suite locally (classic mode): all pass, including the
generic compat sweep that also exercises `signbit`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
