Tim Armstrong has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/11317


Change subject: IMPALA-7483: abort stuck impalad/catalogd on JVM deadlock
......................................................................

IMPALA-7483: abort stuck impalad/catalogd on JVM deadlock

The polling interval can be adjusted with
--jvm_deadlock_detector_interval_s. Setting the interval to <= 0
disables the hang check. I don't think this should ever be needed but I
wanted to keep it around in case we ever see false positive deadlock and
need a way to disable the hang detection.

After studying the OpenJDK code I concluded that the deadlock detection
cost O(# java threads) and had to execute at a VM safepoint so will
block progress of Java threads but not native threads. This seems cheap
enough to execute at one minute intervals without impacting perfomance
measurably.

Testing:
Modified code to replace findDeadlockedThreads() with
getAllThreadIds() to induce the failure. Confirmed that it generated a
core and dumped stacks to impalad.ERROR.

Ran exhaustive tests, mostly to completion except for hitting
IMPALA-7488.

I'm open to ideas about how to write an automated test for this and/or
whether it's worth it - it might be possible to add an impalad debug
flag that deliberately induces a deadlock or write a backend death
test, but I don't know if it's worth the cost of adding and maintaining
complex test infra.

Change-Id: If46f879e7ab9eeead0c6e9a09844e1356012898d
---
M be/src/util/jni-util.cc
M fe/src/main/java/org/apache/impala/common/JniUtil.java
M fe/src/main/java/org/apache/impala/util/JvmPauseMonitor.java
3 files changed, 52 insertions(+), 8 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/17/11317/1
--
To view, visit http://gerrit.cloudera.org:8080/11317
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: If46f879e7ab9eeead0c6e9a09844e1356012898d
Gerrit-Change-Number: 11317
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong <[email protected]>

Reply via email to