This is an automated email from the ASF dual-hosted git repository.
ruifengz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 60bf6817d862 [SPARK-46262][PS] Enable test for `np.left_shift` for
Pandas-on-Spark object
60bf6817d862 is described below
commit 60bf6817d862a8daac629156280996f7dfd07ba7
Author: Haejoon Lee <[email protected]>
AuthorDate: Tue Dec 5 18:33:04 2023 +0800
[SPARK-46262][PS] Enable test for `np.left_shift` for Pandas-on-Spark object
### What changes were proposed in this pull request?
This PR proposes to enable test for `np.left_shift` for Pandas-on-Spark
object.
### Why are the changes needed?
Because now we completely support the PyArrow>=4.0.0, so the behavior
difference within PyArrow<0.15 will not appear.
So we can enable the test since the behavior always same within the
supported version as below:
```python
>>> pdf = pd.DataFrame({"A": [1, 2, 3, 4]})
>>> psdf = ps.DataFrame({"A": [1, 2, 3, 4]})
>>> np.left_shift(pdf, 1)
A
0 2
1 4
2 6
3 8
>>> np.left_shift(psdf, 1)
A
0 2
1 4
2 6
3 8
```
### Does this PR introduce _any_ user-facing change?
No, this just enables the test case.
### How was this patch tested?
The existing CI especially `NumPyCompatTests` should pass.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #44178 from itholic/enable_np_left_shift.
Authored-by: Haejoon Lee <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
---
python/pyspark/pandas/tests/test_numpy_compat.py | 4 ----
1 file changed, 4 deletions(-)
diff --git a/python/pyspark/pandas/tests/test_numpy_compat.py
b/python/pyspark/pandas/tests/test_numpy_compat.py
index af3dcdf9f71f..986083d0d3af 100644
--- a/python/pyspark/pandas/tests/test_numpy_compat.py
+++ b/python/pyspark/pandas/tests/test_numpy_compat.py
@@ -41,10 +41,6 @@ class NumPyCompatTestsMixin:
"log1p", # flaky
"modf",
"floor_divide", # flaky
- # Results seem inconsistent in a different version of, I (Hyukjin)
suspect, PyArrow.
- # From PyArrow 0.15, seems it returns the correct results via PySpark.
Probably we
- # can enable it later when Koalas switches to PyArrow 0.15 completely.
- "left_shift",
]
@property
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
