This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new cf439259a76d [MINOR][PS][DOC] Correct the null-handling comment on the
numpy signbit mapping
cf439259a76d is described below
commit cf439259a76d87fc3c72c2ed03506889a4c57097
Author: Spenser Sun <[email protected]>
AuthorDate: Tue Aug 25 08:41:04 2026 +0800
[MINOR][PS][DOC] Correct the null-handling comment on the numpy signbit
mapping
### What changes were proposed in this pull request?
Comment-only: corrects and completes the null-handling comment on the
`signbit` entry in `unary_np_spark_mappings`, which named fewer cases than
actually reach its floating-null branch.
While fixing it I looked again at the case the comment documents as
unmatched, and I think it can be fixed. `from_pandas` sends both a `NaN` and a
genuine missing value to Spark as a null, so the expression cannot tell which
of the two it has:
- **`float64` / `float32` (default):** a null is always a `NaN`, so the
current answer (`False`) is right.
- **`Float64` / `Float32` (nullable):** a null is an `<NA>` and should
propagate, but it is treated as a `NaN` too -- `np.signbit` on `pd.Series([1.0,
None, -2.0], dtype="Float64")` returns `[False, False, True]` where pandas
returns `[False, <NA>, True]`.
**Proposed fix:** choose the answer for a null from the pandas dtype rather
than from `F.typeof(c)`. `typeof` cannot separate the two dtypes because both
map to `double`, but the dtype is still available in
`maybe_dispatch_ufunc_to_spark_func` before the Column is built, and it settles
the question exactly: a null in a nullable float column can only be an `<NA>`
(propagate), and in a default float column only a `NaN` (`False`). The entry
itself then reduces to the sign test and the `t [...]
**Out of this fix's reach:** the `NaN` *sign*. Spark cannot expose it (`NaN
< 0` is false, and the string cast yields `"NaN"`), so that half of the comment
stands regardless.
Would value your input on whether this is worth pursuing -- happy to do it
under a separate JIRA, or to fold it in here.
### Why are the changes needed?
The comment is the only place this behavior is written down, and it named
fewer cases than land in that branch -- a reader trusting it would expect a
nullable `Float32` `<NA>` to propagate, and a negative `NaN` to be reported as
negative. Neither is true.
### Does this PR introduce _any_ user-facing change?
No. Comment-only.
### How was this patch tested?
No tests added -- the diff is comment-only, so behavior is unchanged. Ran
`ruff check` / `ruff format --check` on the changed file and
`test_numpy_compat.py` (`test_np_signbit` passes).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
Closes #58263 from Spenserrrr/numpy-signbit-null-comment.
Authored-by: Spenser Sun <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
(cherry picked from commit 168a1b5feca5e40989eb103544dac18a6dc4d267)
Signed-off-by: Ruifeng Zheng <[email protected]>
---
python/pyspark/pandas/numpy_compat.py | 14 +++++++++-----
1 file changed, 9 insertions(+), 5 deletions(-)
diff --git a/python/pyspark/pandas/numpy_compat.py
b/python/pyspark/pandas/numpy_compat.py
index c27aa0225e3f..55a12aae853d 100644
--- a/python/pyspark/pandas/numpy_compat.py
+++ b/python/pyspark/pandas/numpy_compat.py
@@ -89,11 +89,15 @@ unary_np_spark_mappings = {
"rint": lambda c: F.rint(c.cast("double")),
"sign": F.signum,
"signbit": lambda c: F.when(
- # A genuine <NA> from a nullable dtype (e.g. Int64) arrives as a
non-double
- # null and must propagate. A NaN from a default (numpy-backed) dtype
arrives
- # as a double null and must map to False (np.signbit(nan) is False);
it falls
- # through to otherwise(False) below. A nullable Float64 <NA> is also a
double
- # null, indistinguishable from a NaN after from_pandas, so it cannot
propagate.
+ # A genuine <NA> from a nullable dtype (e.g. Int64) arrives as a
non-floating null
+ # and must propagate. A NaN from a default (numpy-backed) dtype
arrives as a floating
+ # null and must map to False (np.signbit(nan) is False); it falls
through to
+ # otherwise(False) below. Two cases this expression cannot match,
seeing only the Spark
+ # value and not the pandas dtype: a nullable float dtype's <NA>
(Float32 or Float64) is
+ # also a floating null, indistinguishable from a NaN after
from_pandas, so it reads False
+ # instead of propagating; and the sign of a NaN never reaches here
(from_pandas nulls a
+ # NaN, and a NaN computed in Spark arrives as +NaN), so a negative NaN
reads False where
+ # np.signbit reports True.
c.isNull() & ~F.typeof(c).isin("float", "double"),
F.lit(None).cast("boolean"),
)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
