This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new b5a9e1fee37 [SPARK-41454][PYTHON] Support Python 3.11
b5a9e1fee37 is described below
commit b5a9e1fee374b5d1e86021c7b49f18c5bb1b986e
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Dec 8 16:32:37 2022 -0800
[SPARK-41454][PYTHON] Support Python 3.11
### What changes were proposed in this pull request?
This PR aims to support Python 3.11.
### Why are the changes needed?
Python 3.11 is the newest major release of the Python programming language,
and it contains many new features and optimizations and Python 3.11.1 is the
latest version.
- 2022-12-03 https://www.python.org/downloads/release/python-3111/
And, Spark is affected by one API removal (deprecated at 3.9 and removed at
3.11). Since this is handled by conditionally, there is no regression at the
old Python versions.
- https://bugs.python.org/issue40465
### Does this PR introduce _any_ user-facing change?
No, previsouly, this is not supported.
### How was this patch tested?
Manually run the following. Note that this is tested without optional
dependencies.
```
$ python/run-tests.py --python-executables python3.11
Will test against the following Python executables: ['python3.11']
Will test the following Python modules: ['pyspark-connect', 'pyspark-core',
'pyspark-ml', 'pyspark-mllib', 'pyspark-pandas', 'pyspark-pandas-slow',
'pyspark-resource', 'pyspark-sql', 'pyspark-streaming']
python3.11 python_implementation is CPython
python3.11 version is: Python 3.11.1
Starting test(python3.11): pyspark.ml.tests.test_evaluation (temp output:
/Users/dongjoon/APACHE/spark-merge/python/target/ff09022a-f3d3-413b-b15d-261c40d5b048/python3.11__pyspark.ml.tests.test_evaluation__wh9c4y5l.log)
...
Finished test(python3.11): pyspark.sql.streaming.readwriter (88s)
Tests passed in 1138 seconds
...
Skipped tests in pyspark.tests.test_worker with python3.11:
test_memory_limit
(pyspark.tests.test_worker.WorkerMemoryTest.test_memory_limit) ... skipped
"Memory limit feature in Python worker is dependent on Python's 'resource'
module on Linux; however, not found or not on Linux."
```
Closes #38987 from dongjoon-hyun/SPARK-41454.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
python/pyspark/shuffle.py | 9 ++++++---
python/setup.py | 1 +
2 files changed, 7 insertions(+), 3 deletions(-)
diff --git a/python/pyspark/shuffle.py b/python/pyspark/shuffle.py
index 35c3397de50..da03110c321 100644
--- a/python/pyspark/shuffle.py
+++ b/python/pyspark/shuffle.py
@@ -78,9 +78,12 @@ def _get_local_dirs(sub):
path = os.environ.get("SPARK_LOCAL_DIRS", "/tmp")
dirs = path.split(",")
if len(dirs) > 1:
- # different order in different processes and instances
- rnd = random.Random(os.getpid() + id(dirs))
- random.shuffle(dirs, rnd.random)
+ if sys.version_info < (3, 11):
+ # different order in different processes and instances
+ rnd = random.Random(os.getpid() + id(dirs))
+ random.shuffle(dirs, rnd.random)
+ else:
+ random.shuffle(dirs)
return [os.path.join(d, "python", str(os.getpid()), sub) for d in dirs]
diff --git a/python/setup.py b/python/setup.py
index af102f23083..65db3912efe 100755
--- a/python/setup.py
+++ b/python/setup.py
@@ -282,6 +282,7 @@ try:
'Programming Language :: Python :: 3.8',
'Programming Language :: Python :: 3.9',
'Programming Language :: Python :: 3.10',
+ 'Programming Language :: Python :: 3.11',
'Programming Language :: Python :: Implementation :: CPython',
'Programming Language :: Python :: Implementation :: PyPy',
'Typing :: Typed'],
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
