This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch branch-3.2 in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.2 by this push: new d0fd5fc7e75 [SPARK-39273][PS][TESTS] Make PandasOnSparkTestCase inherit ReusedSQLTestCase d0fd5fc7e75 is described below commit d0fd5fc7e75ccb5c80292cd1058895e1942b75eb Author: Hyukjin Kwon <gurwls...@apache.org> AuthorDate: Wed May 25 09:56:30 2022 +0900 [SPARK-39273][PS][TESTS] Make PandasOnSparkTestCase inherit ReusedSQLTestCase ### What changes were proposed in this pull request? This PR proposes to make `PandasOnSparkTestCase` inherit `ReusedSQLTestCase`. ### Why are the changes needed? We don't need this: ```python classmethod def tearDownClass(cls): # We don't stop Spark session to reuse across all tests. # The Spark session will be started and stopped at PyTest session level. # Please see pyspark/pandas/conftest.py. pass ``` anymore in Apache Spark. This has existed to speed up the tests when the codes are in Koalas repository where the tests run sequentially in single process. In Apache Spark, we run in multiple processes, and we don't need this anymore. ### Does this PR introduce _any_ user-facing change? No, test-only. ### How was this patch tested? Existing CI should test it out. Closes #36652 from HyukjinKwon/SPARK-39273. Authored-by: Hyukjin Kwon <gurwls...@apache.org> Signed-off-by: Hyukjin Kwon <gurwls...@apache.org> (cherry picked from commit a6dd6076d708713d11585bf7f3401d522ea48822) Signed-off-by: Hyukjin Kwon <gurwls...@apache.org> --- python/pyspark/testing/pandasutils.py | 17 ++++------------- 1 file changed, 4 insertions(+), 13 deletions(-) diff --git a/python/pyspark/testing/pandasutils.py b/python/pyspark/testing/pandasutils.py index e957a0f1fbb..df687e265f1 100644 --- a/python/pyspark/testing/pandasutils.py +++ b/python/pyspark/testing/pandasutils.py @@ -18,7 +18,6 @@ import functools import shutil import tempfile -import unittest import warnings from contextlib import contextmanager from distutils.version import LooseVersion @@ -32,9 +31,8 @@ from pyspark import pandas as ps from pyspark.pandas.frame import DataFrame from pyspark.pandas.indexes import Index from pyspark.pandas.series import Series -from pyspark.pandas.utils import default_session, SPARK_CONF_ARROW_ENABLED -from pyspark.testing.sqlutils import SQLTestUtils - +from pyspark.pandas.utils import SPARK_CONF_ARROW_ENABLED +from pyspark.testing.sqlutils import ReusedSQLTestCase tabulate_requirement_message = None try: @@ -61,19 +59,12 @@ except ImportError as e: have_plotly = plotly_requirement_message is None -class PandasOnSparkTestCase(unittest.TestCase, SQLTestUtils): +class PandasOnSparkTestCase(ReusedSQLTestCase): @classmethod def setUpClass(cls): - cls.spark = default_session() + super(PandasOnSparkTestCase, cls).setUpClass() cls.spark.conf.set(SPARK_CONF_ARROW_ENABLED, True) - @classmethod - def tearDownClass(cls): - # We don't stop Spark session to reuse across all tests. - # The Spark session will be started and stopped at PyTest session level. - # Please see pyspark/pandas/conftest.py. - pass - def assertPandasEqual(self, left, right, check_exact=True): if isinstance(left, pd.DataFrame) and isinstance(right, pd.DataFrame): try: --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org