This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 09802865fd7 [SPARK-45053][PYTHON][MINOR] Log improvement in python version mismatch 09802865fd7 is described below commit 09802865fd75947142611e0b73f3c6fa072640ee Author: Wei Liu <wei....@databricks.com> AuthorDate: Mon Sep 4 09:50:54 2023 +0900 [SPARK-45053][PYTHON][MINOR] Log improvement in python version mismatch ### What changes were proposed in this pull request? Before: ``` pyspark.errors.exceptions.base.PySparkRuntimeError: [PYTHON_VERSION_MISMATCH] Python in worker has different version (3, 9) than that in driver 3.10, PySpark cannot run with different minor versions. ``` After: ``` pyspark.errors.exceptions.base.PySparkRuntimeError: [PYTHON_VERSION_MISMATCH] Python in worker has different version: 3.9 than that in driver: 3.10, PySpark cannot run with different minor versions. ``` ### Why are the changes needed? A little more easier to understand the error ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? No need ### Was this patch authored or co-authored using generative AI tooling? No Closes #42776 from WweiL/SPARK-45053-minor-log-improve. Authored-by: Wei Liu <wei....@databricks.com> Signed-off-by: Hyukjin Kwon <gurwls...@apache.org> --- python/pyspark/errors/error_classes.py | 2 +- python/pyspark/worker_util.py | 5 +++-- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/python/pyspark/errors/error_classes.py b/python/pyspark/errors/error_classes.py index 74f52c416e9..c98e9feb610 100644 --- a/python/pyspark/errors/error_classes.py +++ b/python/pyspark/errors/error_classes.py @@ -644,7 +644,7 @@ ERROR_CLASSES_JSON = """ }, "PYTHON_VERSION_MISMATCH" : { "message" : [ - "Python in worker has different version <worker_version> than that in driver <driver_version>, PySpark cannot run with different minor versions.", + "Python in worker has different version: <worker_version> than that in driver: <driver_version>, PySpark cannot run with different minor versions.", "Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set." ] }, diff --git a/python/pyspark/worker_util.py b/python/pyspark/worker_util.py index 9f6d46c6211..722713b6f54 100644 --- a/python/pyspark/worker_util.py +++ b/python/pyspark/worker_util.py @@ -70,11 +70,12 @@ def check_python_version(infile: IO) -> None: Check the Python version between the running process and the one used to serialize the command. """ version = utf8_deserializer.loads(infile) - if version != "%d.%d" % sys.version_info[:2]: + worker_version = "%d.%d" % sys.version_info[:2] + if version != worker_version: raise PySparkRuntimeError( error_class="PYTHON_VERSION_MISMATCH", message_parameters={ - "worker_version": str(sys.version_info[:2]), + "worker_version": worker_version, "driver_version": str(version), }, ) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org