Hello Tianyi Wang, Philip Zeyliger, Todd Lipcon, Impala Public Jenkins, Vuk
Ercegovac,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10998
to look at the new patch set (#6).
Change subject: IMPALA-6857: Add Jvm pause/GC Monitor utility and expose JMX
metrics
......................................................................
IMPALA-6857: Add Jvm pause/GC Monitor utility and expose JMX metrics
Pause monitor:
=============
This commit adds a stripped down version of Hadoop's JvmPauseMonitor
class (https://bit.ly/2O6qSwm) . The core implementaion is borrowed
from hadoop-common project and the hadoop dependencies are removed.
- Removed dependency on AbstractService.
- Not relying on Hadoop's Configuration object for reading confs.
- Switched to Guava's implementation of Stopwatch.
This utility class can detect both GC/non-GC pauses. In case of GC
pauses, the GC metrics during the pause period are logged.
Sample Output:
=============
Detected pause in JVM or host machine (eg GC): pause of approximately
2356ms
GC pool 'PS MarkSweep' had collection(s): count=1 time=2241ms
GC pool 'PS Scavenge' had collection(s): count=3 time=352ms
Detected pause in JVM or host machine (eg GC): pause of approximately
1964ms
GC pool 'PS MarkSweep' had collection(s): count=1 time=2082ms
GC pool 'PS Scavenge' had collection(s): count=1 time=251ms
Detected pause in JVM or host machine (eg GC): pause of approximately
2120ms
GC pool 'PS MarkSweep' had collection(s): count=1 time=2454ms
Detected pause in JVM or host machine (eg GC): pause of approximately
2238ms
GC pool 'PS MarkSweep' had collection(s): count=5 time=13464ms
Detected pause in JVM or host machine (eg GC): pause of approximately
2233ms
GC pool 'PS MarkSweep' had collection(s): count=1 time=2733ms
JMX Metrics:
============
JMX metrics are now emmitted for Impala and Catalog JVMs at the web end
point /jmx.
- Impalad: http(s)://<impalad-host>:25000/jmx
- Catalogd: http(s)://<catalogd-host>:25020/jmx
A text only version (without HTML formatting) of the JMX metrics can be
obtained by passing 'raw' parameter to the /jmx endpoint. For ex:
http://<impalad-host>:25000/jmx?raw
This is useful for downstream tools that want to consume and process
this output.
Misc:
====
Renamed JvmMetric -> JvmMemoryMetric to make the intent more clear. It
doesn't relate to the functionality of the patch in anyway.
Testing:
=======
- Tested it manually with kill -SIGSTOP/-SIGCONT <pid>. Made sure that
the non-GC JVM pauses are logged.
- This class' functionality is tested manually by invoking it's main()
- Injected a memory leak into the Catalog server code and made sure the
GC is detected.
Change-Id: I30d897b7e063846ad6d8f88243e2c04264da0341
---
M be/src/common/init.cc
M be/src/util/default-path-handlers.cc
M be/src/util/jni-util.cc
M be/src/util/jni-util.h
M be/src/util/memory-metrics.cc
M be/src/util/memory-metrics.h
M be/src/util/metrics-test.cc
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/common/JniUtil.java
A fe/src/main/java/org/apache/impala/util/JMXJsonUtil.java
A fe/src/main/java/org/apache/impala/util/JvmPauseMonitor.java
A fe/src/test/java/org/apache/impala/util/JniUtilTest.java
M tests/webserver/test_web_pages.py
13 files changed, 731 insertions(+), 73 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/10998/6
--
To view, visit http://gerrit.cloudera.org:8080/10998
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I30d897b7e063846ad6d8f88243e2c04264da0341
Gerrit-Change-Number: 10998
Gerrit-PatchSet: 6
Gerrit-Owner: Bharath Vissapragada <[email protected]>
Gerrit-Reviewer: Bharath Vissapragada <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Philip Zeyliger <[email protected]>
Gerrit-Reviewer: Tianyi Wang <[email protected]>
Gerrit-Reviewer: Todd Lipcon <[email protected]>
Gerrit-Reviewer: Vuk Ercegovac <[email protected]>
