[
https://issues.apache.org/jira/browse/SPARK-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174019#comment-14174019
]
Andrew Or commented on SPARK-3957:
----------------------------------
Yeah my understanding is that broadcast blocks aren't reported to the driver
(and it makes sense to not report them because the driver is the one who
initiated the broadcast in the first place). The source of the broadcast info
we want to display is in the BlockManager of each executor, and we need to get
this to the driver somehow. We could add some periodic reporting but that opens
another channel between the driver and the executors. There is an ongoing
effort to do something similar for task metrics
https://github.com/apache/spark/pull/2087, so maybe we can piggyback this
information on the heartbeats there.
Also I believe this is a duplicate of an old issue SPARK-1761, though this one
contains more information so let's keep this one open. I will close the other
one in favor of this.
> Broadcast variable memory usage not reflected in UI
> ---------------------------------------------------
>
> Key: SPARK-3957
> URL: https://issues.apache.org/jira/browse/SPARK-3957
> Project: Spark
> Issue Type: Bug
> Components: Block Manager, Web UI
> Affects Versions: 1.0.2, 1.1.0
> Reporter: Shivaram Venkataraman
> Assignee: Nan Zhu
>
> Memory used by broadcast variables are not reflected in the memory usage
> reported in the WebUI. For example, the executors tab shows memory used in
> each executor but this number doesn't include memory used by broadcast
> variables. Similarly the storage tab only shows list of rdds cached and how
> much memory they use.
> We should add a separate column / tab for broadcast variables to make it
> easier to debug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
