On Tue, 27 Oct 2026 15:20:15 GMT, Stefan Karlsson <[email protected]> wrote:
> When running our CI tests we have a failure handler registered that will try > to capture the state of the process when we hit a timeout. It is important to > get thread dumps when the process hangs, and to make sure that we get one as > fast as possible we have recently updated the failure handler to start by > taking an initial Java thread dump. > > That all works well for Java-level deadlocks and hangs. However, it doesn't > work well when it is the JVM that is hanging. When that happens, the > requested Java thread dump will likely be blocked and time out. > > We have two distinct scenarios: > > 1) The JVM is truly deadlocked trying to reach a global safepoint, or there's > a deadlock inside a safepoint operation. > > jstack and many jcmd actions will hang and time out, one at a time. This > takes minutes and after that we will proceed to use the native tools to > inspect the process and dump native stack traces. This scenario is "fine" in > the sense that we probably will get to the native dumping of stack traces > eventually. > > 2) The JVM is not deadlocked, but something is either taking a very long time > to reach a safepoint, or there's a safepoint operation that is taking a very > long time, like a problematic GC verification pass. > > In situations like this, the thread dumping and other jcmds will also hang > and time out, one at a time, until the long operation completes and the test > can continue to run. When that happens, a few of the jcmds could complete, > the test could also complete, and the process could move on and exit. If this > happens, the failure handler doesn't get to the native stack trace dumping, > which is crucial when debugging JVM hangs. > > My proposal is that we make a slight adjustment to the test failure handler > and make it so that it starts the process inspection by taking one native > thread dump, before it moves on to take the Java thread dump. > > It's worth noting that the JVM process is frozen when we take the native > thread dump, so the Java code is not running ahead while it is performed. > This should minimize the risk that we'll miss important information in the > Java thread dump. This is in contrast to the situation we have when we take > the Java thread dump, where the JVM can continue running up until the thread > dump safepoint operation, and some threads will even run non-Java code > concurrently with the Java thread dumper. > > After this change the processes.html shows something like this: > > - test_processes > - 5337 > - initial-native-stack > <The initial native thread... This pull request has now been integrated. Changeset: c196c01d Author: Stefan Karlsson <[email protected]> URL: https://git.openjdk.org/jdk/commit/c196c01d45f7a218f8f2ef6f894910214c74d237 Stats: 76 lines in 6 files changed: 48 ins; 15 del; 13 mod 8395909: Make the test failure_handler take an early native thread stack dump Reviewed-by: jpai, lmesnik ------------- PR: https://git.openjdk.org/jdk/pull/33591
