GitHub user rezasafi opened a pull request:
https://github.com/apache/spark/pull/22612
[SPARK-24958] Add executors' process tree total memory information to
heartbeat signals.
Spark executors' process tree total memory information can be really
useful. Currently such information are not available. The goal of this PR is to
compute such information for each executor, add these information to the
heartbeat signals, and compute the peaks at the driver.
This PR is tested by running the current unit tests and also some added
ones. I have also tested this on our internal cluster and have verified the
results.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rezasafi/spark ptreememory2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22612.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22612
----
commit 3f8321af6f0ae77115f404bb8d15a619c93b5028
Author: Reza Safi <rezasafi@...>
Date: 2018-07-26T21:33:52Z
Integration of ProcessTreeMetrics with PR 21221
Some improvements in integration
Integration with the unit tests of the upstream open PR
Fix an isuue with memory info computation.
Fix scalastyle errors
Some changes to address comments
(cherry picked from commit 04875b8c8ee2f181103331c2cc9544b75b854db6)
commit cd16a75f0d8823d8fd4bfe6bd720b75470398fc4
Author: Reza Safi <rezasafi@...>
Date: 2018-08-07T16:03:46Z
Changing the position of ptree and also make the computation configurable
and some improvments
(cherry picked from commit 29a44c75309d13d071af29d5968f767fba7003d6)
commit 94c2b048603bd4e153473c591fa91fb72a7557d3
Author: Reza Safi <rezasafi@...>
Date: 2018-08-08T21:05:50Z
Seperate metrics for jvm, python and others and update the tests
(cherry picked from commit 36714270d57e3151472363ff2e2c14ab6b88bdea)
commit 062f5d7367f5d50839985325abe6f99fd436a57a
Author: Reza Safi <rezasafi@...>
Date: 2018-09-25T05:02:48Z
Update JsonProtocolSuite
commit 245221d7bc67a06a583b296d23f68ecacc520b34
Author: Reza Safi <rezasafi@...>
Date: 2018-10-02T05:53:45Z
[SPARK-24958] Add executors' process tree total memory information to
heartbeat signals.
Spark executors' process tree total memory information can be really
useful. Currently such information are not available. The goal of this PR is to
compute such information for each executor, add these information to the
heartbeat signals, and compute the peaks at the driver.
This PR is tested by running the current unit tests and also some added
ones. I have also tested this on our internal cluster and have verified the
results.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
