[
https://issues.apache.org/jira/browse/SPARK-7379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15037312#comment-15037312
]
Xusen Yin commented on SPARK-7379:
----------------------------------
[~mengxr] If you look at here:
http://python-future.org/stdlib_incompatibilities.html#array-array, the problem
you pointed above can be solved via the future package.
{code}
#sudo pip3 install future
#python 3.4
>>> import pickle
>>> from future.utils import native_str_to_bytes
>>> pickle.loads(native_str_to_bytes("(dp0\nS'a'\np1\nI1\ns."))
{'a': 1}
>>> pickle.loads(b'\x80\x03}q\x00X\x01\x00\x00\x00aq\x01K\x01s.')
{'a': 1}
{code}
But I am not sure whether the solution could solve the issue.
> pickle.loads expects a string instead of bytes in Python 3.
> -----------------------------------------------------------
>
> Key: SPARK-7379
> URL: https://issues.apache.org/jira/browse/SPARK-7379
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 1.4.0
> Reporter: Xiangrui Meng
> Assignee: Davies Liu
>
> In PickleSerializer, we call pickle.loads in Python 3. However, the input obj
> could be bytes, which works in Python 2 but not 3.
> The error message is
> {code}
> File
> "/home/jenkins/workspace/SparkPullRequestBuilder@3/python/pyspark/serializers.py",
> line 418, in loads
> return pickle.loads(obj, encoding=encoding)
> TypeError: must be a unicode character, not bytes
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
