Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11317
Change subject: IMPALA-7483: abort stuck impalad/catalogd on JVM deadlock ...................................................................... IMPALA-7483: abort stuck impalad/catalogd on JVM deadlock The polling interval can be adjusted with --jvm_deadlock_detector_interval_s. Setting the interval to <= 0 disables the hang check. I don't think this should ever be needed but I wanted to keep it around in case we ever see false positive deadlock and need a way to disable the hang detection. After studying the OpenJDK code I concluded that the deadlock detection cost O(# java threads) and had to execute at a VM safepoint so will block progress of Java threads but not native threads. This seems cheap enough to execute at one minute intervals without impacting perfomance measurably. Testing: Modified code to replace findDeadlockedThreads() with getAllThreadIds() to induce the failure. Confirmed that it generated a core and dumped stacks to impalad.ERROR. Ran exhaustive tests, mostly to completion except for hitting IMPALA-7488. I'm open to ideas about how to write an automated test for this and/or whether it's worth it - it might be possible to add an impalad debug flag that deliberately induces a deadlock or write a backend death test, but I don't know if it's worth the cost of adding and maintaining complex test infra. Change-Id: If46f879e7ab9eeead0c6e9a09844e1356012898d --- M be/src/util/jni-util.cc M fe/src/main/java/org/apache/impala/common/JniUtil.java M fe/src/main/java/org/apache/impala/util/JvmPauseMonitor.java 3 files changed, 52 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/17/11317/1 -- To view, visit http://gerrit.cloudera.org:8080/11317 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: If46f879e7ab9eeead0c6e9a09844e1356012898d Gerrit-Change-Number: 11317 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong <[email protected]>
