Alex Angelini created SPARK-4603:
------------------------------------
Summary: EOF when broadcasting a dict with an empty string value.
Key: SPARK-4603
URL: https://issues.apache.org/jira/browse/SPARK-4603
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 1.1.0
Environment: OSX 10.10
Reporter: Alex Angelini
Steps to reproduce:
1. Broadcast {'a': ''}
2. Try to read the value of the broadcast
{code}
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 1.3.0-SNAPSHOT
/_/
Using Python version 2.7.8 (default, Oct 19 2014 16:02:00)
SparkContext available as sc.
In [1]: sc
Out[1]: <pyspark.context.SparkContext at 0x1077e4710>
In [2]: b = sc.broadcast({'a': ''})
In [3]: b.value
---------------------------------------------------------------------------
EOFError Traceback (most recent call last)
<ipython-input-3-59996048613f> in <module>()
----> 1 b.value
/Users/alexangelini/src/starscream/spark/current/python/pyspark/broadcast.pyc
in value(self)
75 if not hasattr(self, "_value") and self.path is not None:
76 ser = LargeObjectSerializer()
---> 77 self._value = ser.load_stream(open(self.path)).next()
78 return self._value
79
/Users/alexangelini/src/starscream/spark/current/python/pyspark/serializers.pyc
in load_stream(self, stream)
615 yield value
616 elif type == 'P':
--> 617 yield cPickle.load(stream)
618 else:
619 raise ValueError("unknown type: %s" % type)
EOFError:
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
