Shivaram Venkataraman created SPARK-4031:
--------------------------------------------
Summary: Read broadcast variables on use
Key: SPARK-4031
URL: https://issues.apache.org/jira/browse/SPARK-4031
Project: Spark
Issue Type: Bug
Components: Block Manager, Spark Core
Reporter: Shivaram Venkataraman
Assignee: Shivaram Venkataraman
This is a proposal to change the broadcast variable implementations in Spark to
only read values when they are used rather than on deserializing.
This change will be very helpful (and in our use cases required) for complex
applications which have a large number of broadcast variables. For example if
broadcast variables are class members, they are captured in closures even when
they are not used.
We could also consider cleaning closures more aggressively, but that might be a
more complex change.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
