GitHub user LantaoJin opened a pull request:
https://github.com/apache/spark/pull/22874
[WIP][SPARK-25865][CORE] Add GC information to ExecutorMetrics
## What changes were proposed in this pull request?
This PR is opened on top of the PR for #22612 since it import an interface
of `getMetricSet` in `ExecutorMetricType`.
Only memory usage without GC information could not help us to determinate
the proper settings of memory. Add basic GC information to ExecutorMetrics
interface.
## How was this patch tested?
Existing UTs.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/LantaoJin/spark SPARK-25865
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22874.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22874
----
commit 3f8321af6f0ae77115f404bb8d15a619c93b5028
Author: Reza Safi <rezasafi@...>
Date: 2018-07-26T21:33:52Z
Integration of ProcessTreeMetrics with PR 21221
Some improvements in integration
Integration with the unit tests of the upstream open PR
Fix an isuue with memory info computation.
Fix scalastyle errors
Some changes to address comments
(cherry picked from commit 04875b8c8ee2f181103331c2cc9544b75b854db6)
commit cd16a75f0d8823d8fd4bfe6bd720b75470398fc4
Author: Reza Safi <rezasafi@...>
Date: 2018-08-07T16:03:46Z
Changing the position of ptree and also make the computation configurable
and some improvments
(cherry picked from commit 29a44c75309d13d071af29d5968f767fba7003d6)
commit 94c2b048603bd4e153473c591fa91fb72a7557d3
Author: Reza Safi <rezasafi@...>
Date: 2018-08-08T21:05:50Z
Seperate metrics for jvm, python and others and update the tests
(cherry picked from commit 36714270d57e3151472363ff2e2c14ab6b88bdea)
commit 062f5d7367f5d50839985325abe6f99fd436a57a
Author: Reza Safi <rezasafi@...>
Date: 2018-09-25T05:02:48Z
Update JsonProtocolSuite
commit 245221d7bc67a06a583b296d23f68ecacc520b34
Author: Reza Safi <rezasafi@...>
Date: 2018-10-02T05:53:45Z
[SPARK-24958] Add executors' process tree total memory information to
heartbeat signals.
Spark executors' process tree total memory information can be really
useful. Currently such information are not available. The goal of this PR is to
compute such information for each executor, add these information to the
heartbeat signals, and compute the peaks at the driver.
This PR is tested by running the current unit tests and also some added
ones. I have also tested this on our internal cluster and have verified the
results.
commit c72be03d72998a357c6f241e0f6e2fa751c9e1c8
Author: Reza Safi <rezasafi@...>
Date: 2018-10-03T21:30:16Z
Adressing most of Imran's comments
commit 8f3c9382ec5f6c0d729c689053cee88f7234ad36
Author: Reza Safi <rezasafi@...>
Date: 2018-10-03T23:02:49Z
Fixing the scala style and some minor comments
commit f2dca27359e2e51ac53df0a8fae2c3cd0b966946
Author: Reza Safi <rezasafi@...>
Date: 2018-10-04T03:12:02Z
Removing types from the definitions where ever possible
commit a9f924c5943d6ed45e38a1c5aadd07045adbe138
Author: Reza Safi <rezasafi@...>
Date: 2018-10-05T22:01:48Z
Using Utils methods when possible or use ProcessBuilder
commit a11e3a267b78cf5a7e42190893f36e24e2aad2d4
Author: Reza Safi <rezasafi@...>
Date: 2018-10-05T22:58:44Z
make use of Utils.trywithresources
commit 34ad625d60a72ec270f8e57280a8eb9016eba6e4
Author: Reza Safi <rezasafi@...>
Date: 2018-10-09T15:40:06Z
Changing ExecutorMericType and ExecutorMetrics to use a map instead of an
array of metrics
commit 415f976642d7cabc7b32a6b511c5e099ab60377d
Author: Reza Safi <rezasafi@...>
Date: 2018-10-10T19:02:09Z
Changing ExecutorMetric to use array instead of a map
commit 067b81d24de7999afe5b9660e89d9a2e41de6d21
Author: Reza Safi <rezasafi@...>
Date: 2018-10-10T23:54:10Z
A small cosmetic change
commit 18ee4ad3829c3ee637d45f73c4e182b78c7661fb
Author: Reza Safi <rezasafi@...>
Date: 2018-10-17T20:23:56Z
Merge branch 'master' of https://github.com/apache/spark into ptreememory2
commit 7f7ed2bdf5740bd2c4ae8cf2090ba7f016ffb023
Author: Reza Safi <rezasafi@...>
Date: 2018-10-23T04:34:42Z
Applying latest review commments. Using Arrays instead of Map for returning
set of metrics.
commit a049454fd0511e9ed8ff06a7e887c9bfbbcdb7d8
Author: LantaoJin <jinlantao@...>
Date: 2018-10-29T09:11:46Z
[WIP][SPARK-25865][CORE] Add GC information to ExecutorMetrics
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
