Hello Bharath Vissapragada, Philip Zeyliger, Todd Lipcon, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/11317
to look at the new patch set (#3).
Change subject: IMPALA-7483: abort stuck impalad/catalogd on JVM deadlock
......................................................................
IMPALA-7483: abort stuck impalad/catalogd on JVM deadlock
The polling interval can be adjusted with
--jvm_deadlock_detector_interval_s. Setting the interval to <= 0
disables the hang check. I don't think this should ever be needed but I
wanted to keep it around in case we ever see false positive deadlock and
need a way to disable the hang detection.
After studying the OpenJDK code I concluded that the deadlock detection
cost O(# java threads) and had to execute at a VM safepoint so will
block progress of Java threads but not native threads. This seems cheap
enough to execute at one minute intervals without impacting perfomance
measurably.
Testing:
Modified code to replace findDeadlockedThreads() with
getAllThreadIds() to induce the failure in impalad and catalogd.
Confirmed that it generated a core and dumped stacks to impalad.ERROR.
Added manual testing in JvmPauseMonitor.main() that induces a deadlock:
CLASSPATH=$CLASSPATH:fe/target/impala-frontend-0.1-SNAPSHOT.jar \
java org.apache.impala.util.JvmPauseMonitor deadlock
Ran exhaustive tests, mostly to completion except for hitting
IMPALA-7488.
I'm open to ideas about how to write an automated test for this and/or
whether it's worth it - it might be possible to add an impalad debug
flag that deliberately induces a deadlock or write a backend death
test, but I don't know if it's worth the cost of adding and maintaining
complex test infra.
Change-Id: If46f879e7ab9eeead0c6e9a09844e1356012898d
---
M be/src/util/jni-util.cc
M fe/src/main/java/org/apache/impala/common/JniUtil.java
M fe/src/main/java/org/apache/impala/util/JvmPauseMonitor.java
3 files changed, 114 insertions(+), 14 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/17/11317/3
--
To view, visit http://gerrit.cloudera.org:8080/11317
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If46f879e7ab9eeead0c6e9a09844e1356012898d
Gerrit-Change-Number: 11317
Gerrit-PatchSet: 3
Gerrit-Owner: Tim Armstrong <[email protected]>
Gerrit-Reviewer: Bharath Vissapragada <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Philip Zeyliger <[email protected]>
Gerrit-Reviewer: Tim Armstrong <[email protected]>
Gerrit-Reviewer: Todd Lipcon <[email protected]>