I experienced the below two cases when unpersisting or destroying broadcast
variables in pyspark. But the same works good in spark scala shell. Any clue
why this happens ? Is it a bug in pyspark?

***Case 1:***
    >>> b1 = sc.broadcast([1,2,3])
    >>> b1.value
    [1, 2, 3]
    >>> b1.destroy()
    >>> b1.value
    [1, 2, 3]
I can still access the value in driver.

***Case 2:***
    >>> b = sc.broadcast([1,2,3])
    >>> b.destroy()
    >>> b.value
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
line 109, in value
        self._value = self.load(self._path)
line 95, in load
        with open(path, 'rb', 1 << 20) as f:
    IOError: [Errno 2] No such file or directory:

Rather i should get a message something similar to "Attempted to use
broadcast variable after it was destroyed"

Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Reply via email to