Github user JeetKunDoug commented on the issue:
https://github.com/apache/spark/pull/21322
@cloud-fan So I see what's been messing with me - it's the whole Broadcast
variable cache/week reference thing. I originally wrote this internally for
Spark 1.6.3 (still supporting some older Spark installations), which didn't
have this concept. I'll take another look at how we may be able to better
handle the executor-wide cache in a more coherent way (pun intended) and get
back to you. You may be right that we will have to depend on a combination of a
finalizer + autocloseable object handled more directly in `Broadcast#destroy`
for the happy-path case where we're not running into memory pressure on the
MemoryStore that is causing the variable to churn.... I just hate having to
depend on a finalizer to do this if there's a way around it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
