[
https://issues.apache.org/jira/browse/SPARK-21157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16372789#comment-16372789
]
assia ydroudj commented on SPARK-21157:
---------------------------------------
this paragraph is well interesting for me: "Perhaps we could
show a trend line of memory usage allowing a user to see at which stage
boundary memory usage
increased." Is it done please ????
how can we get memory reporting values gathered in a json file or other formats
to further exploitation?
thanks
> Report Total Memory Used by Spark Executors
> -------------------------------------------
>
> Key: SPARK-21157
> URL: https://issues.apache.org/jira/browse/SPARK-21157
> Project: Spark
> Issue Type: Improvement
> Components: Input/Output
> Affects Versions: 2.1.1
> Reporter: Jose Soltren
> Priority: Major
> Attachments: TotalMemoryReportingDesignDoc.pdf
>
>
> Building on some of the core ideas of SPARK-9103, this JIRA proposes tracking
> total memory used by Spark executors, and a means of broadcasting,
> aggregating, and reporting memory usage data in the Spark UI.
> Here, "total memory used" refers to memory usage that is visible outside of
> Spark, to an external observer such as YARN, Mesos, or the operating system.
> The goal of this enhancement is to give Spark users more information about
> how Spark clusters are using memory. Total memory will include non-Spark JVM
> memory and all off-heap memory.
> Please consult the attached design document for further details.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
