GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/13920
[SPARK-16166][Core] Also take off-heap memory usage into consideration in
log and webui display
## What changes were proposed in this pull request?
Currently in the log and UI display, only on-heap storage memory is
calculated and displayed,
```
16/06/27 13:41:52 INFO MemoryStore: Block rdd_5_0 stored as values in
memory (estimated size 17.8 KB, free 665.9 MB)
```
<img width="1232" alt="untitled"
src="https://cloud.githubusercontent.com/assets/850797/16369960/53fb614e-3c6e-11e6-8fa3-7ffe65abcb49.png";>
With [SPARK-13992](https://issues.apache.org/jira/browse/SPARK-13992)
off-heap memory is supported for data persistence, so here change to also take
off-heap storage memory into consideration.
## How was this patch tested?
Unit test and local verification.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jerryshao/apache-spark SPARK-16166
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13920.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13920
----
commit 58fbc00b06fa612d3fa2927ea204d8fbd8484627
Author: jerryshao <[email protected]>
Date: 2016-06-23T06:54:43Z
honor offHeapStorageMemoryPool in log and web ui display
commit e3194bde7078c850181c0c2686cecaf0a0705fc2
Author: jerryshao <[email protected]>
Date: 2016-06-23T08:32:26Z
Fix unit test failure
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
![https://cloud.githubusercontent.com/assets/850797/16369960/53fb614e-3c6e-11e6-8fa3-7ffe65abcb49.png"](https://cloud.githubusercontent.com/assets/850797/16369960/53fb614e-3c6e-11e6-8fa3-7ffe65abcb49.png"){kind=link}