GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/14617
[SPARK-17019][Core] Expose on-heap and off-heap memory usage various places
## What changes were proposed in this pull request?
With [SPARK-13992](https://issues.apache.org/jira/browse/SPARK-13992),
Spark supports persisting data into off-heap memory, but the usage of on-heap
and off-heap memory is not exposed currently, it is not so convenient for user
to monitor and profile, so here propose to expose off-heap memory as well as
on-heap memory usage in various places:
1. Spark UI's executor page will display both on-heap and off-heap memory
usage.
2. REST request returns both on-heap and off-heap memory.
3. Also this can be gotten from MetricsSystem.
4. Last this usage can be obtained programmatically from SparkListener.
Attach the UI changes:
Backward compatibility is also considered for event-log and REST API. Old
event log can still be replayed with off-heap usage displayed as 0. For REST
API, only adds the new fields, so JSON backward compatibility can still be kept.
## How was this patch tested?
Unit test added and manual verification.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jerryshao/apache-spark SPARK-17019
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14617.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14617
----
commit a5e9d46aedf6de47f1e93de0afd8b5d913f2f36e
Author: jerryshao <[email protected]>
Date: 2016-08-12T02:51:12Z
Expose on-heap and off-heap memory usage in UI, REST API and SparkListener
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
