[
https://issues.apache.org/jira/browse/SPARK-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173954#comment-14173954
]
Shivaram Venkataraman commented on SPARK-3957:
----------------------------------------------
I think it needs to be tracked in the Block Manager -- However we also need to
track this on a per-executor basis and not just at the driver. Right now AFAIK,
executors do not report new broadcast blocks to the master to reduce
communication. However we could add broadcast blocks to some periodic report.
[~andrewor] might know more.
> Broadcast variable memory usage not reflected in UI
> ---------------------------------------------------
>
> Key: SPARK-3957
> URL: https://issues.apache.org/jira/browse/SPARK-3957
> Project: Spark
> Issue Type: Bug
> Components: Block Manager, Web UI
> Affects Versions: 1.0.2, 1.1.0
> Reporter: Shivaram Venkataraman
> Assignee: Nan Zhu
>
> Memory used by broadcast variables are not reflected in the memory usage
> reported in the WebUI. For example, the executors tab shows memory used in
> each executor but this number doesn't include memory used by broadcast
> variables. Similarly the storage tab only shows list of rdds cached and how
> much memory they use.
> We should add a separate column / tab for broadcast variables to make it
> easier to debug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
