Are there limits to SA's ability to produce a stack trace for a thread?

[Chris Plummer]

Hello,

[Sorry this email got kind of long. To cut to the chase, I want to know if there are times where it is acceptable for SA to not be able to produce a stack trace for a thread. Details below if you are interested.]

We have a number of SA tests that request a thread dump, look for a specific symbol in the thread dump, and fail if the symbol is not found. Normally what they are looking for is LingeredApp.main() which should be in the stack trace of the main thread. ClhsdbJstack.java is one such test. It expects the main thread to look like:

"main" #1 prio=5 tid=0x000001d6301de800 nid=0x3258 waiting on condition [0x0000007fc1dff000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   JavaThread state: _thread_blocked
 - java.lang.Thread.sleep(long) @bci=0, pc=0x000001d640d0f417, Method*=0x00000008000e8898 (Interpreted frame)
 - jdk.test.lib.apps.LingeredApp.main(java.lang.String[]) @bci=54, line=499, pc=0x000001d640d0a1b3, Method*=0x000001d658673ba0 (Interpreted frame)

But sometimes all it gets is:

"main" #1 prio=5 tid=0x00007fab2e802000 nid=0x2303 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   JavaThread state: _thread_in_java

This results in the test failing because it does not find LingeredApp.main in the output. The state for the passing case is always _thread_blocked and for the failing case _thread_in_java. This has been reported by the following CR:

[1] JDK-8242411 - serviceability/sa/ClhsdbCDSJstackPrintAll.java fails with Test ERROR java.lang.RuntimeException: 'LingeredApp.main' missing from stdout/stderr

After starting, LingeredApp.main sits in a loop:

            while (Files.exists(path)) {
                // Touch the lock to indicate our readiness
                setLastModified(theLockFileName, epoch());
                Thread.sleep(spinDelay);
            }

So it's basically waiting for the lock file to be deleted. By default spinDelay is 1 second. I suspected the issue I was seeing was due to asking for the thread dump when not blocked on the sleep(), so I changed spingDelay to 1ms. That made this missing stack trace issue much easier to reproduce, plus a several other bugs that are filed, but normally rarely reproduce:

[2] JDK-8231634 - SA stack walking fails with "illegal bci"
[3] JDK-8240781 - serviceability/sa/ClhsdbJdis.java fails with "java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1"
[4] JDK-8211923 - [Testbug] serviceability/sa/ClhsdbFindPC.java ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1
[5] JDK-8204994 - SA might fail to attach to process with "Windbg Error: WaitForEvent failed"

The "illegal bci" failure I haven't looked into much, but is likely an SA bug due to SA having issues with (and probably making assumptions about) the state of the stack.

The two ArrayIndexOutOfBoundsException bugs are dups. They fail because the stack trace of the main thread is missing, and some String splitting logic in the test therefore fails and produces the ArrayIndexOutOfBoundsException.

I'm not sure about the "WaitForEvent failed". It could be unrelated.

I can probably make these all go away buy having Lingered.main() spawn a helper thread to do the above loop in. That would keep the main thread stable (blocked on a Thread.join). However, it also would hide some issues(like the "illegal bci" failure).

The main reason for the email is to ask what are the expectations of SA's ability to dump a thread's stack trace. Is it expected that sometimes the thread will be in a state that prevents dumping the stack? I know for example that the reason we sometimes don't see a stack is because thread.getLastJavaVFrameDbg() is returning null. Basically SA throws up its hands and says "I can't do it"? Is that acceptable in some cases.

thanks,

Chris

[1] https://bugs.openjdk.java.net/browse/JDK-8242411
[2] https://bugs.openjdk.java.net/browse/JDK-8231634
[3] https://bugs.openjdk.java.net/browse/JDK-8240781
[4] https://bugs.openjdk.java.net/browse/JDK-8211923
[5] https://bugs.openjdk.java.net/browse/JDK-8204994