GitHub user HeartSaVioR opened a pull request:
https://github.com/apache/spark/pull/21469
[SPARK-24441][SS] Expose total size of states in HDFSBackedStateStoreâ¦
â¦Provider
## What changes were proposed in this pull request?
This patch exposes the estimation of size of cache (loadedMaps) in
HDFSBackedStateStoreProvider as a custom metric of StateStore. While it refers
loadedMaps directly, there would be only one StateStoreWriter which refers a
StateStoreProvider, so the value is not exposed as well as being aggregated
multiple times. Current state metrics are safe to aggregate for the same reason.
## How was this patch tested?
Tested manually. Below is the snapshot of UI page which is reflected by the
patch:
<img width="596" alt="screen shot 2018-05-31 at 10 13 23 pm"
src="https://user-images.githubusercontent.com/1317309/40788976-4ad93d8c-652c-11e8-88f1-337be5162588.png";>
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HeartSaVioR/spark SPARK-24441
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21469.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21469
----
commit dc11338650842a246a4bce9280d130607ceca281
Author: Jungtaek Lim <kabhwan@...>
Date: 2018-05-31T14:38:00Z
[SPARK-24441][SS] Expose total size of states in
HDFSBackedStateStoreProvider
* expose estimation of size of cache (loadMaps) in
HDFSBackedStateStoreProvider
* as a custom metric of StateStore
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
![https://user-images.githubusercontent.com/1317309/40788976-4ad93d8c-652c-11e8-88f1-337be5162588.png"](https://user-images.githubusercontent.com/1317309/40788976-4ad93d8c-652c-11e8-88f1-337be5162588.png"){kind=link}