GitHub user liyezhang556520 opened a pull request:
https://github.com/apache/spark/pull/7753
[SPARK-9104][CORE][WIP] expose Netty network layer memory used in shuffle
read part
This is a sub-task of
[SPARK-9103](https://issues.apache.org/jira/browse/SPARK-9103), we'd like to
expose the memory usage for spark running time, this is the first step to
expose the netty buffer used both with on-heap and off-heap memory. Also the
metrics are showed on WebUI. In this PR, a new web Tab name `Memory` is added.
Which is used to show the memory usage of each executors (can be in more
details in future). the screenshot is like the following:
This is WIP because the exposed metrics are not recorded into eventlog yet
and also unit tests are not added and some situations are not handled
(executors added, removed, failed, etc.). One important thing is to get some
feedback from the community, any comments and thoughts are really appreciated.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/liyezhang556520/spark spark-9104-draft
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/7753.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #7753
----
commit 1917729e2f405daf404550a1873d0e857fc30742
Author: Zhang, Liye <[email protected]>
Date: 2015-07-21T06:42:38Z
SPARK-9212 upgrade Netty version to 4.0.29.Final
commit e6e7947440e3b05d164f3228b23d361052935f22
Author: Zhang, Liye <[email protected]>
Date: 2015-07-21T09:47:28Z
Merge remote-tracking branch 'apache/master'
commit ea1864e1d59b9d5089451932735c6a3be270633f
Author: Zhang, Liye <[email protected]>
Date: 2015-07-22T05:51:15Z
initial test, functionality test, test data fetch from netty
commit edda857c3d763347037a44f898feb005827de4f0
Author: Zhang, Liye <[email protected]>
Date: 2015-07-22T07:00:54Z
change the place to start metric collection
commit e487e4daaf8bd67d1a4fa8da02ffa508d08c284f
Author: Zhang, Liye <[email protected]>
Date: 2015-07-22T07:21:10Z
start the metric collector back to server initial func
commit 9cc65c352c42554d8f57f611e6892f847f26478c
Author: Zhang, Liye <[email protected]>
Date: 2015-07-22T07:36:23Z
start metric collection when starting block manager
commit 467dedc3531471e556584159046f93f127415bfd
Author: Zhang, Liye <[email protected]>
Date: 2015-07-22T07:38:41Z
add executor as the parameter
commit 8f26d9feed48a2fc10d1f2fe782bb2db1bbc90dd
Author: Zhang, Liye <[email protected]>
Date: 2015-07-22T09:06:54Z
chang int to long, avoid overflow
commit 9d7d7d8b4713a4ab11326722e99c4ed79cf349e8
Author: Zhang, Liye <[email protected]>
Date: 2015-07-26T15:51:06Z
add executor metrics
commit d25bdfd87178220e5c97c95db7de6ce3cc4cdfd7
Author: Zhang, Liye <[email protected]>
Date: 2015-07-27T08:49:58Z
initial webUI with memory metrics
commit 3474d162e9236d89cdeea683d1e329ea17a89013
Author: Zhang, Liye <[email protected]>
Date: 2015-07-27T09:22:40Z
code refinement
commit de9a5edaf11e2a9a1e1791ce9eee8089f9d2e9d9
Author: Zhang, Liye <[email protected]>
Date: 2015-07-29T16:22:11Z
collect metrics in heartbeat and add timestamp for max mem size
commit 17e5b978618a5a6adfa3ff621e37eeecaa0b2b0c
Author: Zhang, Liye <[email protected]>
Date: 2015-07-29T17:18:15Z
avoid un-serializable clock for executor metrics
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
