Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/11317 )
Change subject: IMPALA-7483: abort stuck impalad/catalogd on JVM deadlock ...................................................................... Patch Set 2: (5 comments) http://gerrit.cloudera.org:8080/#/c/11317/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11317/2//COMMIT_MSG@22 PS2, Line 22: findDeadlockedThreads Maybe worth injecting a deadlock locally and see if findDeadlockedThreads() works as expected too (if you have not done that already). http://gerrit.cloudera.org:8080/#/c/11317/2/be/src/util/jni-util.cc File be/src/util/jni-util.cc: http://gerrit.cloudera.org:8080/#/c/11317/2/be/src/util/jni-util.cc@28 PS2, Line 28: 60 IMO, configuring it something like 300s is reasonable too since VM seems to be executing a safepoint for this. Also since we see deadlocks very rarely. http://gerrit.cloudera.org:8080/#/c/11317/2/be/src/util/jni-util.cc@29 PS2, Line 29: "Interval between JVM deadlock checks. If set to 0 or a negative value, deadlock " nit: mark it (Advanced)? http://gerrit.cloudera.org:8080/#/c/11317/2/fe/src/main/java/org/apache/impala/util/JvmPauseMonitor.java File fe/src/main/java/org/apache/impala/util/JvmPauseMonitor.java: http://gerrit.cloudera.org:8080/#/c/11317/2/fe/src/main/java/org/apache/impala/util/JvmPauseMonitor.java@176 PS2, Line 176: return; not related to your change but I just noticed this, can we log this as an error with exception trace so that we know if the pause monitor exits. http://gerrit.cloudera.org:8080/#/c/11317/2/fe/src/main/java/org/apache/impala/util/JvmPauseMonitor.java@217 PS2, Line 217: LOG.fatal("Aborting because of deadlocked threads in JVM."); Should we dump the entire thread stacks (not just the deadlocked ones) and other JVM state, say JMX output at this point? (gives heap usage etc. if needed). Also any other stuff like /metrics etc? -- To view, visit http://gerrit.cloudera.org:8080/11317 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If46f879e7ab9eeead0c6e9a09844e1356012898d Gerrit-Change-Number: 11317 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong <[email protected]> Gerrit-Reviewer: Bharath Vissapragada <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Philip Zeyliger <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]> Gerrit-Reviewer: Todd Lipcon <[email protected]> Gerrit-Comment-Date: Fri, 24 Aug 2018 20:47:08 +0000 Gerrit-HasComments: Yes
