[ https://issues.apache.org/jira/browse/SPARK-29536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17213558#comment-17213558 ]
Dongjoon Hyun commented on SPARK-29536: --------------------------------------- Hi, [~hyukjin.kwon]. Apache Spark 2.4.7 also fails. I will update the affected version. {code} $ current_pyspark Python 3.8.5 (default, Sep 10 2020, 11:46:28) [Clang 11.0.0 (clang-1100.0.33.16)] on darwin Type "help", "copyright", "credits" or "license" for more information. Traceback (most recent call last): File "/Users/dongjoon/APACHE/spark-release/spark-2.4.7-bin-hadoop2.7/python/pyspark/shell.py", line 31, in <module> from pyspark import SparkConf File "/Users/dongjoon/APACHE/spark-release/spark-2.4.7-bin-hadoop2.7/python/pyspark/__init__.py", line 51, in <module> from pyspark.context import SparkContext File "/Users/dongjoon/APACHE/spark-release/spark-2.4.7-bin-hadoop2.7/python/pyspark/context.py", line 31, in <module> from pyspark import accumulators File "/Users/dongjoon/APACHE/spark-release/spark-2.4.7-bin-hadoop2.7/python/pyspark/accumulators.py", line 97, in <module> from pyspark.serializers import read_int, PickleSerializer File "/Users/dongjoon/APACHE/spark-release/spark-2.4.7-bin-hadoop2.7/python/pyspark/serializers.py", line 72, in <module> from pyspark import cloudpickle File "/Users/dongjoon/APACHE/spark-release/spark-2.4.7-bin-hadoop2.7/python/pyspark/cloudpickle.py", line 145, in <module> _cell_set_template_code = _make_cell_set_template_code() File "/Users/dongjoon/APACHE/spark-release/spark-2.4.7-bin-hadoop2.7/python/pyspark/cloudpickle.py", line 126, in _make_cell_set_template_code return types.CodeType( TypeError: an integer is required (got type bytes) >>> {code} > PySpark does not work with Python 3.8.0 > --------------------------------------- > > Key: SPARK-29536 > URL: https://issues.apache.org/jira/browse/SPARK-29536 > Project: Spark > Issue Type: Test > Components: PySpark > Affects Versions: 3.0.0 > Reporter: Hyukjin Kwon > Assignee: Hyukjin Kwon > Priority: Critical > Fix For: 3.0.0 > > > You open a shell and run arbitrary codes: > {code} > File "/.../3.8/lib/python3.8/runpy.py", line 183, in _run_module_as_main > mod_name, mod_spec, code = _get_module_details(mod_name, _Error) > File "/.../3.8/lib/python3.8/runpy.py", line 109, in _get_module_details > __import__(pkg_name) > File /.../workspace/forked/spark/python/pyspark/__init__.py", line 51, in > <module> > from pyspark.context import SparkContext > File "/.../spark/python/pyspark/context.py", line 31, in <module> > from pyspark import accumulators > File "/.../python/pyspark/accumulators.py", line 97, in <module> > from pyspark.serializers import read_int, PickleSerializer > File "/.../python/pyspark/serializers.py", line 71, in <module> > from pyspark import cloudpickle > File "/.../python/pyspark/cloudpickle.py", line 152, in <module> > _cell_set_template_code = _make_cell_set_template_code() > File "/.../spark/python/pyspark/cloudpickle.py", line 133, in > _make_cell_set_template_code > return types.CodeType( > TypeError: an integer is required (got type bytes) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org