[
https://issues.apache.org/jira/browse/SPARK-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Rosen resolved SPARK-2954.
-------------------------------
Resolution: Fixed
Fix Version/s: 1.1.0
> PySpark MLlib serialization tests fail on Python 2.6
> ----------------------------------------------------
>
> Key: SPARK-2954
> URL: https://issues.apache.org/jira/browse/SPARK-2954
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 1.1.0
> Reporter: Josh Rosen
> Assignee: Josh Rosen
> Fix For: 1.1.0
>
>
> The PySpark MLlib tests currently fail on Python 2.6 due to problems
> unpacking data from bytearray using struct.unpack:
> {code}
> **********************************************************************
> File "pyspark/mllib/_common.py", line 181, in __main__._deserialize_double
> Failed example:
> _deserialize_double(_serialize_double(1L)) == 1.0
> Exception raised:
> Traceback (most recent call last):
> File
> "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py",
> line 1253, in __run
> compileflags, 1) in test.globs
> File "<doctest __main__._deserialize_double[4]>", line 1, in <module>
> _deserialize_double(_serialize_double(1L)) == 1.0
> File "pyspark/mllib/_common.py", line 194, in _deserialize_double
> return struct.unpack("d", ba[offset:])[0]
> error: unpack requires a string argument of length 8
> **********************************************************************
> File "pyspark/mllib/_common.py", line 184, in __main__._deserialize_double
> Failed example:
> _deserialize_double(_serialize_double(sys.float_info.max)) == x
> Exception raised:
> Traceback (most recent call last):
> File
> "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py",
> line 1253, in __run
> compileflags, 1) in test.globs
> File "<doctest __main__._deserialize_double[6]>", line 1, in <module>
> _deserialize_double(_serialize_double(sys.float_info.max)) == x
> File "pyspark/mllib/_common.py", line 194, in _deserialize_double
> return struct.unpack("d", ba[offset:])[0]
> error: unpack requires a string argument of length 8
> **********************************************************************
> File "pyspark/mllib/_common.py", line 187, in __main__._deserialize_double
> Failed example:
> _deserialize_double(_serialize_double(sys.float_info.max)) == y
> Exception raised:
> Traceback (most recent call last):
> File
> "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py",
> line 1253, in __run
> compileflags, 1) in test.globs
> File "<doctest __main__._deserialize_double[8]>", line 1, in <module>
> _deserialize_double(_serialize_double(sys.float_info.max)) == y
> File "pyspark/mllib/_common.py", line 194, in _deserialize_double
> return struct.unpack("d", ba[offset:])[0]
> error: unpack requires a string argument of length 8
> **********************************************************************
> {code}
> It looks like one solution is to wrap the {{bytearray}} with {{buffer()}}:
> http://stackoverflow.com/a/15467046/590203
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
