[
https://issues.apache.org/jira/browse/SPARK-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14178068#comment-14178068
]
Patrick Wendell edited comment on SPARK-4030 at 10/21/14 7:17 AM:
------------------------------------------------------------------
Hey Shivaram - IIRC we made this private to protect users from shooting
themselves in the foot and trying to use a broadcast variable that has been
destroy()'d, potentially by another thread or somewhere difficult to track
down. If we expose this it would be good to test it thoroughly (I think we put
some logic in there around this already thinking that we might open it up
later). Also, it might be good to capture the callsite of the destroy call and
echo it back to users if they attempt to use an invalid broadcast variable.
I also notice we have this terminology around "valid" in the Broadcast class,
it might make more sense to change that to "destroyed" to make it more explicit
- that's just a minor clean-up though and we could do it in a different class.
was (Author: pwendell):
Hey Shivaram - IIRC we made this private to protect users from shooting
themselves in the foot and trying to use a broadcast variable that has been
destroy()'d, potentially by another thread or somewhere difficult to track
down. If we expose this it would be good to test it thoroughly (I think we put
some logic in there around this already thinking that we might open it up
later). Also, it might be good to capture the callsite of the destroy call and
echo it back to users if they attempt to use an invalid broadcast variable.
> `destroy` method in Broadcast should be public
> ----------------------------------------------
>
> Key: SPARK-4030
> URL: https://issues.apache.org/jira/browse/SPARK-4030
> Project: Spark
> Issue Type: Improvement
> Components: Block Manager, Spark Core
> Affects Versions: 1.1.0, 1.2.0
> Reporter: Shivaram Venkataraman
>
> The destroy method in Broadcast.scala
> [https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/broadcast/Broadcast.scala#L91]
> is right now marked as private[spark]
> This prevents long-running applications from cleaning up memory used by
> broadcast variables on the driver. Also as broadcast variables are always
> created with persistence MEMORY_DISK, this slows down jobs when old broadcast
> variables are flushed to disk.
> Making `destroy` public can help applications control the lifetime.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
