This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 09802865fd7 [SPARK-45053][PYTHON][MINOR] Log improvement in python
version mismatch
09802865fd7 is described below
commit 09802865fd75947142611e0b73f3c6fa072640ee
Author: Wei Liu <[email protected]>
AuthorDate: Mon Sep 4 09:50:54 2023 +0900
[SPARK-45053][PYTHON][MINOR] Log improvement in python version mismatch
### What changes were proposed in this pull request?
Before:
```
pyspark.errors.exceptions.base.PySparkRuntimeError:
[PYTHON_VERSION_MISMATCH] Python in worker has different version (3, 9) than
that in driver 3.10, PySpark cannot run with different minor versions.
```
After:
```
pyspark.errors.exceptions.base.PySparkRuntimeError:
[PYTHON_VERSION_MISMATCH] Python in worker has different version: 3.9 than that
in driver: 3.10, PySpark cannot run with different minor versions.
```
### Why are the changes needed?
A little more easier to understand the error
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
No need
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #42776 from WweiL/SPARK-45053-minor-log-improve.
Authored-by: Wei Liu <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
python/pyspark/errors/error_classes.py | 2 +-
python/pyspark/worker_util.py | 5 +++--
2 files changed, 4 insertions(+), 3 deletions(-)
diff --git a/python/pyspark/errors/error_classes.py
b/python/pyspark/errors/error_classes.py
index 74f52c416e9..c98e9feb610 100644
--- a/python/pyspark/errors/error_classes.py
+++ b/python/pyspark/errors/error_classes.py
@@ -644,7 +644,7 @@ ERROR_CLASSES_JSON = """
},
"PYTHON_VERSION_MISMATCH" : {
"message" : [
- "Python in worker has different version <worker_version> than that in
driver <driver_version>, PySpark cannot run with different minor versions.",
+ "Python in worker has different version: <worker_version> than that in
driver: <driver_version>, PySpark cannot run with different minor versions.",
"Please check environment variables PYSPARK_PYTHON and
PYSPARK_DRIVER_PYTHON are correctly set."
]
},
diff --git a/python/pyspark/worker_util.py b/python/pyspark/worker_util.py
index 9f6d46c6211..722713b6f54 100644
--- a/python/pyspark/worker_util.py
+++ b/python/pyspark/worker_util.py
@@ -70,11 +70,12 @@ def check_python_version(infile: IO) -> None:
Check the Python version between the running process and the one used to
serialize the command.
"""
version = utf8_deserializer.loads(infile)
- if version != "%d.%d" % sys.version_info[:2]:
+ worker_version = "%d.%d" % sys.version_info[:2]
+ if version != worker_version:
raise PySparkRuntimeError(
error_class="PYTHON_VERSION_MISMATCH",
message_parameters={
- "worker_version": str(sys.version_info[:2]),
+ "worker_version": worker_version,
"driver_version": str(version),
},
)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
