This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 5ca3467b8653 [SPARK-47729][PYTHON][TESTS] Get the proper default port for pyspark-connect testcases 5ca3467b8653 is described below commit 5ca3467b86531b971f92c4d9da3ecc2735ae2214 Author: Hyukjin Kwon <gurwls...@apache.org> AuthorDate: Thu Apr 4 07:33:24 2024 -0700 [SPARK-47729][PYTHON][TESTS] Get the proper default port for pyspark-connect testcases ### What changes were proposed in this pull request? This PR proposes to get the proper default port for `pyspark-connect` testcases. ### Why are the changes needed? `pyspark-connect` cannot access to the JVM, so cannot get the randomized port assigned from JVM. ### Does this PR introduce _any_ user-facing change? No, `pyspark-connect` is not published yet, and this is a test-only change. ### How was this patch tested? Manually tested. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #45875 from HyukjinKwon/SPARK-47729. Authored-by: Hyukjin Kwon <gurwls...@apache.org> Signed-off-by: Dongjoon Hyun <dh...@apple.com> --- python/pyspark/sql/connect/client/core.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/python/pyspark/sql/connect/client/core.py b/python/pyspark/sql/connect/client/core.py index dd7fae881aec..b731960bbaf3 100644 --- a/python/pyspark/sql/connect/client/core.py +++ b/python/pyspark/sql/connect/client/core.py @@ -56,6 +56,7 @@ import grpc from google.protobuf import text_format, any_pb2 from google.rpc import error_details_pb2 +from pyspark.util import is_remote_only from pyspark.accumulators import SpecialAccumulatorIds from pyspark.loose_version import LooseVersion from pyspark.version import __version__ @@ -292,7 +293,7 @@ class DefaultChannelBuilder(ChannelBuilder): @staticmethod def default_port() -> int: - if "SPARK_TESTING" in os.environ: + if "SPARK_TESTING" in os.environ and not is_remote_only(): from pyspark.sql.session import SparkSession as PySparkSession # In the case when Spark Connect uses the local mode, it starts the regular Spark --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org