This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new eb327b68ab85 [SPARK-46161][PS][FOLLOWUP] Validate periods type in
DataFrame.diff axis=1 branch
eb327b68ab85 is described below
commit eb327b68ab8571d425ed08d820ae8ccbafabf32f
Author: Yicong Huang <[email protected]>
AuthorDate: Tue Aug 25 08:56:12 2026 +0800
[SPARK-46161][PS][FOLLOWUP] Validate periods type in DataFrame.diff axis=1
branch
### What changes were proposed in this pull request?
This is a follow-up to
[SPARK-46161](https://issues.apache.org/jira/browse/SPARK-46161), which added
`axis=1` support to pandas-on-Spark `DataFrame.diff`. The new `axis=1` branch
skipped the `periods` type validation that the `axis=0` path already enforces
(via `Series._diff`). This PR adds the same guard at the top of the `axis=1`
branch so a non-integer `periods` raises the established `TypeError` instead of
a raw Python list-index error.
### Why are the changes needed?
The `axis=0` path validates `periods` in `Series._diff` and raises
`TypeError("periods should be an int; however, got [...]")`. The `axis=1`
branch computed `prev_idx = i - periods` and then indexed
`column_labels[prev_idx]`, so a non-integer `periods` leaked an unhelpful
Python error rather than the documented one. For example, before this change:
```python
>>> import pyspark.pandas as ps
>>> psdf = ps.DataFrame({"a": [1, 2, 3], "b": [1, 2, 3], "c": [1, 2, 3]})
>>> psdf.diff(1.5, axis=1)
TypeError: list indices must be integers or slices, not float
```
After this change it matches the `axis=0` behavior:
```python
>>> psdf.diff(1.5, axis=1)
TypeError: periods should be an int; however, got [float]
```
### Does this PR introduce _any_ user-facing change?
Yes, but only within the unreleased `axis=1` support added by SPARK-46161.
Passing a non-integer `periods` with `axis=1` now raises `TypeError: periods
should be an int; however, got [...]` instead of `TypeError: list indices must
be integers or slices, not float`. There is no change compared to released
Spark versions, and valid calls are unaffected.
### How was this patch tested?
Added a negative assertion to `test_diff` in
`python/pyspark/pandas/tests/computation/test_compute.py` covering
`psdf.diff(1.5, axis=1)`, mirroring the existing `axis=0` check.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #58259 from Yicong-Huang/spark-46161-periods-type-validation.
Authored-by: Yicong Huang <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
---
python/pyspark/pandas/frame.py | 4 ++++
python/pyspark/pandas/tests/computation/test_compute.py | 3 +++
2 files changed, 7 insertions(+)
diff --git a/python/pyspark/pandas/frame.py b/python/pyspark/pandas/frame.py
index fc075ba32494..11ea453685a6 100644
--- a/python/pyspark/pandas/frame.py
+++ b/python/pyspark/pandas/frame.py
@@ -5029,6 +5029,10 @@ defaultdict(<class 'list'>, {'col..., 'col...})]
if axis == 0:
return self._apply_series_op(lambda psser: psser._diff(periods),
should_resolve=True)
else:
+ if not isinstance(periods, int):
+ raise TypeError(
+ "periods should be an int; however, got [%s]" %
type(periods).__name__
+ )
column_labels = self._internal.column_labels
data_col_names = self._internal.data_spark_column_names
new_columns: list[PySparkColumn] = []
diff --git a/python/pyspark/pandas/tests/computation/test_compute.py
b/python/pyspark/pandas/tests/computation/test_compute.py
index 924046fe256b..7fb1b72592ed 100644
--- a/python/pyspark/pandas/tests/computation/test_compute.py
+++ b/python/pyspark/pandas/tests/computation/test_compute.py
@@ -204,6 +204,9 @@ class FrameComputeMixin:
self.assert_eq(pdf.diff(periods=2, axis=1), psdf.diff(periods=2,
axis=1))
self.assert_eq(pdf.diff(periods=-1, axis=1), psdf.diff(periods=-1,
axis=1))
+ with self.assertRaisesRegex(TypeError, msg):
+ psdf.diff(1.5, axis=1)
+
# multi-index columns
columns = pd.MultiIndex.from_tuples([("x", "Col1"), ("x", "Col2"),
("y", "Col3")])
pdf.columns = columns
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
