This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 714fc8cd872d [SPARK-48316][PS][CONNECT][TESTS] Fix comments for
SparkFrameMethodsParityTests.test_coalesce and test_repartition
714fc8cd872d is described below
commit 714fc8cd872d6f583a6066e9ddb4a51caa51caf3
Author: Hyukjin Kwon <gurwls...@apache.org>
AuthorDate: Fri May 17 12:09:49 2024 +0900
[SPARK-48316][PS][CONNECT][TESTS] Fix comments for
SparkFrameMethodsParityTests.test_coalesce and test_repartition
### What changes were proposed in this pull request?
This PR proposes to enable `SparkFrameMethodsParityTests.test_coalesce` and
`SparkFrameMethodsParityTests.test_repartition` in Spark Connect by avoiding
RDD usage in the test.
### Why are the changes needed?
To make sure on the test coverage
### Does this PR introduce _any_ user-facing change?
No, test-only.
### How was this patch tested?
CI in this PR.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #46629 from HyukjinKwon/SPARK-48316.
Authored-by: Hyukjin Kwon <gurwls...@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls...@apache.org>
---
python/pyspark/pandas/tests/connect/test_parity_frame_spark.py | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/python/pyspark/pandas/tests/connect/test_parity_frame_spark.py
b/python/pyspark/pandas/tests/connect/test_parity_frame_spark.py
index 24626a9164e8..c3672647b71b 100644
--- a/python/pyspark/pandas/tests/connect/test_parity_frame_spark.py
+++ b/python/pyspark/pandas/tests/connect/test_parity_frame_spark.py
@@ -28,7 +28,9 @@ class SparkFrameMethodsParityTests(
def test_checkpoint(self):
super().test_checkpoint()
- @unittest.skip("Test depends on RDD which is not supported from Spark
Connect.")
+ @unittest.skip(
+ "Test depends on RDD, and cannot use SQL expression due to Catalyst
optimization"
+ )
def test_coalesce(self):
super().test_coalesce()
@@ -36,7 +38,9 @@ class SparkFrameMethodsParityTests(
def test_local_checkpoint(self):
super().test_local_checkpoint()
- @unittest.skip("Test depends on RDD which is not supported from Spark
Connect.")
+ @unittest.skip(
+ "Test depends on RDD, and cannot use SQL expression due to Catalyst
optimization"
+ )
def test_repartition(self):
super().test_repartition()
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
