GitHub user jsoltren opened a pull request:
https://github.com/apache/spark/pull/17762
[SPARK-9103][WIP] Track Netty memory usage - take two
This patch resurrects #7753 by liyezhang556520 and #17625 by jsoltren.
## What changes were proposed in this pull request?
Memory reporting is a much desired feature in Apache Spark. This change
adds reporting for some memory used by netty. We introduce a new
executorMetrics object to track this.
This is a backend only change. No UI is included with this change.
## How was this patch tested?
Ran
https://github.com/jsoltren/spark-demos/blob/master/src/main/scala/org/soltren/spark/examples/ShuffleGroupCount.scala
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jsoltren/spark jsoltren-9103
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17762.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17762
----
commit 1e51c73d340b331f6fcd033635e4468201ba1477
Author: José Hiram Soltren <[email protected]>
Date: 2017-03-28T19:49:38Z
Netty network layer memory usage on webUI
Manual pull of the PR from:
https://github.com/apache/spark/pull/7753/commits
Created with:
git checkout upstream/pr/7753 -- .
A bunch of manual merging happened after this.
commit 543cd88f18210311691c86db30eb6ad8618386b6
Author: José Hiram Soltren <[email protected]>
Date: 2017-04-04T17:53:34Z
more fixups to get things compiling
commit b794b86a37e816938a3172bb0a7aeb0e95ca0ca7
Author: José Hiram Soltren <[email protected]>
Date: 2017-04-04T18:13:46Z
fix JsonProtocolSuite
commit 941eef06587d2b20a63f46c86c3534b88a6d3bf7
Author: José Hiram Soltren <[email protected]>
Date: 2017-04-11T01:56:05Z
Some fixups related to internal code review. More are coming.
commit 72ed41eb449626e06fbed22b3a96dfd3f299e58d
Author: José Hiram Soltren <[email protected]>
Date: 2017-04-11T01:58:54Z
delete stragglers
commit 7cddd6c335a2a961a3b9e8061ad32f57a6a5063a
Author: José Hiram Soltren <[email protected]>
Date: 2017-04-12T19:02:58Z
Doc changes and minor tweaks in preparation for PR
commit 1bb14b476af6d55ce0c53b9b9b40946f92cac1c9
Author: José Hiram Soltren <[email protected]>
Date: 2017-04-12T21:53:43Z
s/Happen Time/Peak Time
commit f3e5704b4db4d9da54ed57c44c9381689b6d682b
Author: José Hiram Soltren <[email protected]>
Date: 2017-04-12T22:11:02Z
fix merge conflict
commit 577d44211888232541a39026850ab7aa7b125e70
Author: José Hiram Soltren <[email protected]>
Date: 2017-04-17T16:02:39Z
Respond to some code review feedback. Make executorMetrics an Option.
commit 0ebcec0f3505e88e37f2acc56a4dda3cf52fb236
Author: José Hiram Soltren <[email protected]>
Date: 2017-04-24T18:36:55Z
Get tests to compile after making ExecutorMetrics an Option
commit f3d8aa57f3b777c54e3e2295df007017ec1058fd
Author: José Hiram Soltren <[email protected]>
Date: 2017-04-24T18:47:04Z
More SQL test fixes
commit 550c2d8bd57a55e2801f9e5569b0425a62558255
Author: José Hiram Soltren <[email protected]>
Date: 2017-04-25T20:06:01Z
Remove UI parts of this change set
Refactor MemoryListener to core from UI.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
