Hi, Spark 1.2.0, standalone, local mode(for test)
Here are several questions on broadcast variable: 1) Where is the broadcast variable cached on executors ? In memory or On disk ? I read somewhere, it was said these variables are stored in spark.local.dir. But I can find any info in Spark 1.2 document. I encountered a problem with broadcast variables. I have a loop in which a broadcast variable is created, after 3 iteration, the used memory increased quickly until the full size, and Spark is blocked, no error message, no exception, just blocked. I would like to make sure whether it is caused by too many broadcast variables, because I did not call unpersist() on each broadcast variable. 2) I find that broadcast variable has destroy() and unpersist() method, what's the difference between them? If a broadcast variable is destroyed, is it removed from where it is stored ? Hao -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Broadcast-variable-questions-tp21292.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
