GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/19160
[SPARK-21934][CORE] Expose Shuffle Netty memory usage to MetricsSystem
## What changes were proposed in this pull request?
This is a followup work of SPARK-9104 to expose the Netty memory usage to
MetricsSystem. Current the shuffle Netty memory usage of
`NettyBlockTransferService` will be exposed, if using external shuffle, then
the Netty memory usage of `ExternalShuffleClient` and `ExternalShuffleService`
will be exposed instead. Currently I don't expose Netty memory usage of
`YarnShuffleService`, because `YarnShuffleService` doesn't have `MetricsSystem`
itself, and is better to connect to Hadoop's MetricsSystem.
## How was this patch tested?
Manually verified in local cluster.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jerryshao/apache-spark SPARK-21934
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19160.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19160
----
commit 04a7ec944b3273fbe9b9bdb6e217814452a1a12c
Author: jerryshao <[email protected]>
Date: 2017-09-07T13:25:39Z
Expose Shuffle Netty memory usage to MetricsSystem
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
