[
https://issues.apache.org/jira/browse/SPARK-9103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16176073#comment-16176073
]
Saisai Shao commented on SPARK-9103:
------------------------------------
[~irashid] I would like to hear your suggestion on displaying Netty memory
usage on web UI and REST API.
Currently with SPARK-21934, we already support exposing Netty memory usage via
MetricsSystem, user could connect metric sink to Graphite or StatsD to get each
executor's metrics information. But we don't have a centralized place to
display the Netty memory usage for all the executors. In the previous PR we
tried to collect back such metrics to driver through heartbeat, and displayed
them on web UI. This somehow seems useful, but actually it duplicates the
functionality of MetricsSystem. So I'm clearly sure if such functionality
(display Netty memory usage on web UI and RESTR API) is worthwhile. Can you
please help suggesting on it? Thanks a lot.
> Tracking spark's memory usage
> -----------------------------
>
> Key: SPARK-9103
> URL: https://issues.apache.org/jira/browse/SPARK-9103
> Project: Spark
> Issue Type: Umbrella
> Components: Spark Core, Web UI
> Reporter: Zhang, Liye
> Attachments: Tracking Spark Memory Usage - Phase 1.pdf
>
>
> Currently spark only provides little memory usage information (RDD cache on
> webUI) for the executors. User have no idea on what is the memory consumption
> when they are running spark applications with a lot of memory used in spark
> executors. Especially when they encounter the OOM, it’s really hard to know
> what is the cause of the problem. So it would be helpful to give out the
> detail memory consumption information for each part of spark, so that user
> can clearly have a picture of where the memory is exactly used.
> The memory usage info to expose should include but not limited to shuffle,
> cache, network, serializer, etc.
> User can optionally choose to open this functionality since this is mainly
> for debugging and tuning.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
