This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 6803bdcc52ef [SPARK-58790][PS][FOLLOWUP] Fix NumPy modf on a DataFrame
with no columns
6803bdcc52ef is described below
commit 6803bdcc52efeaf7b8ab60391a770095c2f65632
Author: Spenser Sun <[email protected]>
AuthorDate: Tue Sep 1 09:08:08 2026 +0800
[SPARK-58790][PS][FOLLOWUP] Fix NumPy modf on a DataFrame with no columns
### What changes were proposed in this pull request?
`DataFrame.__array_ufunc__` now takes the number of outputs from
`ufunc.nout` rather than inspecting the first per-column result, which is what
pandas does (`pandas/core/arraylike.py`). The per-column call moves into a
local helper so the single-output path builds its columns in a single pass
again.
### Why are the changes needed?
A DataFrame with no columns yields no per-column result to inspect, so a
multi-output ufunc took the single-output branch and returned one DataFrame
instead of a two-element tuple:
```python
>>> frac, integ = np.modf(psdf[[]])
ValueError: not enough values to unpack (expected 2, got 0)
```
pandas returns two empty DataFrames here.
### Does this PR introduce _any_ user-facing change?
Yes, within the unreleased branches only. `np.modf` on a DataFrame with no
columns now returns a `(fractional, integral)` tuple as it does for every other
DataFrame, instead of a single DataFrame.
### How was this patch tested?
Extended `test_np_modf` in `NumPyCompatTestsMixin` with a zero-column
DataFrame case, which fails without this change.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
Closes #58427 from Spenserrrr/numpy-modf-zero-column.
Authored-by: Spenser Sun <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
(cherry picked from commit 7f5306b0c90cc4c03acf9eb2ebc32cf1372a0777)
Signed-off-by: Ruifeng Zheng <[email protected]>
---
python/pyspark/pandas/frame.py | 15 +++++++++------
python/pyspark/pandas/tests/test_numpy_compat.py | 7 +++++++
2 files changed, 16 insertions(+), 6 deletions(-)
diff --git a/python/pyspark/pandas/frame.py b/python/pyspark/pandas/frame.py
index 73919101f996..2dbb6f5f46b0 100644
--- a/python/pyspark/pandas/frame.py
+++ b/python/pyspark/pandas/frame.py
@@ -14430,22 +14430,25 @@ defaultdict(<class 'list'>, {'col..., 'col...})]
this = inputs[0]
assert all(inp is this for inp in inputs if isinstance(inp,
DataFrame))
column_labels = this._internal.column_labels
- outputs = [
- ufunc(
+
+ def apply_ufunc(label: Label) -> Any:
+ return ufunc(
*[inp[label] if isinstance(inp, DataFrame) else inp for
inp in inputs], **kwargs
)
- for label in column_labels
- ]
- if outputs and isinstance(outputs[0], tuple):
+
+ # numpy declares the number of outputs on the ufunc itself, so a
frame with no
+ # columns still returns the right number of DataFrames.
+ if getattr(ufunc, "nout", 1) > 1:
# A multi-output ufunc (for example np.modf) yields a
two-element tuple per
# column; regroup into one DataFrame per output, matching
numpy/pandas.
+ outputs = [apply_ufunc(label) for label in column_labels]
first_cols = [out[0].rename(label) for out, label in
zip(outputs, column_labels)]
second_cols = [out[1].rename(label) for out, label in
zip(outputs, column_labels)]
first_df: DataFrame =
DataFrame(this._internal.with_new_columns(first_cols))
second_df: DataFrame =
DataFrame(this._internal.with_new_columns(second_cols))
return first_df, second_df
else:
- applied = [out.rename(label) for out, label in zip(outputs,
column_labels)]
+ applied = [apply_ufunc(label).rename(label) for label in
column_labels]
internal = this._internal.with_new_columns(applied)
return DataFrame(internal)
diff --git a/python/pyspark/pandas/tests/test_numpy_compat.py
b/python/pyspark/pandas/tests/test_numpy_compat.py
index 04dc60181020..13a8594ca011 100644
--- a/python/pyspark/pandas/tests/test_numpy_compat.py
+++ b/python/pyspark/pandas/tests/test_numpy_compat.py
@@ -375,6 +375,13 @@ class NumPyCompatTestsMixin:
self.assert_eq(np.signbit(ps_fractional.to_pandas()),
np.signbit(pd_fractional))
self.assert_eq(np.signbit(ps_integral.to_pandas()),
np.signbit(pd_integral))
+ # A DataFrame with no columns has no per-column result to inspect, so
the number of
+ # outputs comes from the ufunc; pandas returns one empty DataFrame per
output here too.
+ ps_fractional, ps_integral = np.modf(psdf[[]])
+ pd_fractional, pd_integral = np.modf(pdf[[]])
+ self.assert_eq(ps_fractional, pd_fractional)
+ self.assert_eq(ps_integral, pd_integral)
+
# Index input: np.modf returns a tuple of Index objects.
pidx = pd.Index([-3.5, -2.0, -0.5, 0.0, 2.7])
psidx = ps.from_pandas(pidx)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
