On 11/20/2014 05:03 AM, Volker Simonis wrote: > Hy Maynard, > > I've just realized that in your patch the two directory patterns > $(AGENT_SRC_DIR)/sun/jvm/hotspot/runtime/linux_ppc64/*.java and > $(AGENT_SRC_DIR)/sun/jvm/hotspot/runtime/ppc64/*.java are absent from > "make/sa.files". This of course breaks incremental builds. Nice, thank you. I was wondering why incremental build didn't work. I had resorted to using a build script that did the following: rm build/linux-ppc64-normal-server-release/hotspot/linux_ppc64_compiler2/product/../generated/sa-jdi.jar make all JOBS=24 make install
-Maynard > > Regards, > Volker > > > diff -r 6f35dca1949c make/sa.files > --- a/make/sa.files Mon Nov 17 14:47:41 2014 +0100 > +++ b/make/sa.files Thu Nov 20 11:58:52 2014 +0100 > @@ -94,12 +94,14 @@ > $(AGENT_SRC_DIR)/sun/jvm/hotspot/runtime/linux_amd64/*.java \ > $(AGENT_SRC_DIR)/sun/jvm/hotspot/runtime/linux_x86/*.java \ > $(AGENT_SRC_DIR)/sun/jvm/hotspot/runtime/linux_sparc/*.java \ > +$(AGENT_SRC_DIR)/sun/jvm/hotspot/runtime/linux_ppc64/*.java \ > $(AGENT_SRC_DIR)/sun/jvm/hotspot/runtime/posix/*.java \ > $(AGENT_SRC_DIR)/sun/jvm/hotspot/runtime/solaris_amd64/*.java \ > $(AGENT_SRC_DIR)/sun/jvm/hotspot/runtime/solaris_sparc/*.java \ > $(AGENT_SRC_DIR)/sun/jvm/hotspot/runtime/solaris_x86/*.java \ > $(AGENT_SRC_DIR)/sun/jvm/hotspot/runtime/sparc/*.java \ > $(AGENT_SRC_DIR)/sun/jvm/hotspot/runtime/x86/*.java \ > +$(AGENT_SRC_DIR)/sun/jvm/hotspot/runtime/ppc64/*.java \ > $(AGENT_SRC_DIR)/sun/jvm/hotspot/tools/*.java \ > $(AGENT_SRC_DIR)/sun/jvm/hotspot/tools/jcore/*.java \ > $(AGENT_SRC_DIR)/sun/jvm/hotspot/tools/soql/*.java \ > > On Wed, Nov 19, 2014 at 7:33 PM, Volker Simonis > <[email protected]> wrote: >> Hi Maynard, >> >> I just wanted to let you know that I'm still working on fixing the >> bogus entries in the stack trace. I'm pretty sure they are related to >> inlining. If you run your test program with "-XX:+PrintCompilation >> -XX:+PrintInlining -XX:CICompilerCount=1" you'll get the following >> output: >> >> .. >> 10954 5 % test::run_test @ 59 (99 bytes) >> @ 75 java.lang.String::getChars (62 >> bytes) inline (hot) >> @ 58 java.lang.System::arraycopy (0 >> bytes) (intrinsic) >> @ 87 test::get_my_chars (86 bytes) inline >> (hot) >> @ 6 java.lang.String::getChars (62 >> bytes) inline (hot) >> @ 58 java.lang.System::arraycopy (0 >> bytes) (intrinsic) >> @ 38 java.lang.String::<init> (62 >> bytes) inline (hot) >> @ 1 java.lang.Object::<init> (1 >> bytes) inline (hot) >> @ 55 java.util.Arrays::copyOfRange >> (63 bytes) too big >> @ 47 java.lang.StringBuilder::<init> >> (7 bytes) inline (hot) >> @ 52 java.lang.StringBuilder::append >> (8 bytes) executed < MinInliningThreshold times >> @ 57 java.lang.StringBuilder::append >> (8 bytes) executed < MinInliningThreshold times >> @ 62 java.lang.StringBuilder::append >> (8 bytes) executed < MinInliningThreshold times >> @ 65 java.lang.StringBuilder::toString >> (17 bytes) executed < MinInliningThreshold times >> @ 79 java.lang.String::length (6 >> bytes) inline (hot) >> @ 82 java.io.OutputStreamWriter::write >> (11 bytes) executed < MinInliningThreshold times >> .. >> >> The stack trace I get from jstack looks as follows: >> >> Thread 4448: (state = IN_JAVA) >> - test.get_my_chars(int, int, char[], java.lang.String, long) >> @bci=43, line=15 (Compiled frame; information may be imprecise) >> - test.run_test() @bci=87, line=35 (Compiled frame) >> - java.lang.String.getChars(int, int, char[], int) @bci=58, line=814 >> (Compiled frame) >> - test.run_test() @bci=75, line=34 (Compiled frame) >> >> From a system perspective 'test::run_test' is one native frame, >> because 'test::run_test' inlines all the other functions reported >> above. HotSpot has special functionality to detect and walk these >> inlined methods (so called "virtual frames" or "vframe"s). For some >> reason this vframe walking doesn't seem to work in the agent. In gdb, >> when calling "ps()" at the same point where I created the above core >> file I'll get the following stack trace: >> >> (gdb) call ps() >> >> "Executing ps" >> for thread: "main" #1 prio=5 os_prio=0 tid=0x00003fffb0010800 >> nid=0x1160 runnable [0x0000000000000000] >> java.lang.Thread.State: RUNNABLE >> JavaThread state: _thread_in_Java >> Thread: 0x00003fffb0010800 [0x1160] State: _running _has_called_back >> 0 _at_poll_safepoint 0 >> JavaThread state: _thread_in_Java >> >> (guessing starting frame id=0x3fffb66ddc60 based on current fp) >> C frame (sp=0x00003fffb66ddad0 unextended sp=0x00003fffb66ddad0, >> fp=0x00003fffb66ddc60, real_fp=0x00003fffb66ddc60, >> pc=0x000000001000067c) >> 1 - frame( sp=0x00003fffb66ddc60, unextended_sp=0x00003fffb66ddc60, >> fp=0x00003fffb66ddd60, pc=0x00003fffa0159d58) >> test.get_my_chars(test.java:16) >> 2 - frame( sp=0x00003fffb66ddc60, unextended_sp=0x00003fffb66ddc60, >> fp=0x00003fffb66ddd60, pc=0x00003fffa0159d58) >> test.run_test(test.java:35) >> 3 - frame( sp=0x00003fffb66ddd60, unextended_sp=0x00003fffb66ddd60, >> fp=0x00003fffb66dde60, pc=0x00003fffa000f518) >> test.main(test.java:56) >> >> I'll keep you informed once I fixed the problem (I'll also look into >> the .opd issue afterwards). >> >> Regards, >> Volker >> >> >> On Tue, Nov 18, 2014 at 12:20 AM, Maynard Johnson <[email protected]> >> wrote: >>> On 11/17/2014 01:21 PM, Volker Simonis wrote: >>>> On Mon, Nov 17, 2014 at 6:59 PM, Maynard Johnson <[email protected]> >>>> wrote: >>>>> On 11/17/2014 10:20 AM, Volker Simonis wrote: >>>>>> Hi Maynard, >>>>>> >>>>>> I'm currently looking at your changes. At first glance they look good. >>>>>> >>>>>> I could open a simple core file which contained both, interpreted and >>>>>> compiled frames: >>>>>> >>>>>> $ jstack ./images/j2sdk-image/bin/java core.7034 >>>>>> ... >>>>>> Thread 7035: (state = IN_VM) >>>>>> - sun.misc.Unsafe.putAddress(long, long) @bci=0 (Interpreted frame) >>>>>> - Crash.crashIt(sun.misc.Unsafe, int) @bci=10, line=8 (Interpreted >>>>>> frame) >>>>>> - Crash.doIt() @bci=45, line=23 (Compiled frame) >>>>>> - sun.reflect.NativeMethodAccessorImpl.invoke0(java.lang.reflect.Method, >>>>>> java.lang.Object, java.lang.Object[]) @bci=0 (Interpreted frame) >>>>>> - sun.reflect.NativeMethodAccessorImpl.invoke(java.lang.Object, >>>>>> java.lang.Object[]) @bci=100, line=62 (Interpreted frame) >>>>>> - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, >>>>>> java.lang.Object[]) @bci=6, line=43 (Interpreted frame) >>>>>> - java.lang.reflect.Method.invoke(java.lang.Object, >>>>>> java.lang.Object[]) @bci=56, line=498 (Interpreted frame) >>>>>> - Crash.main(java.lang.String[]) @bci=32, line=31 (Interpreted frame) >>>>>> >>>>>> The one thing that doesn't currently work is "jstack -m" (i.e. "mixed >>>>>> mode" for java and native frames). Are you aware of this? >>>>> Hi, Volker, >>>>> Yeah, I knew about this problem and forgot to mention it in my patch >>>>> posting. I started >>>>> looking at it this morning, and so far, I have at least fixed the >>>>> UnmappedAddressException. >>>>> But now I'm getting different results on little endian vs big endian >>>>> ppc64 systems. >>>>> On BE, I either get no symbol names (i.e., "?????") or wrong symbol >>>>> names. On LE, >>>>> I seem to get correct symbol names for the first symbol (either >>>>> __pthread_cond_wait >>>>> or __pthread_cond_timedwait) and the last symbol (start_thread) of each >>>>> stack, but >>>>> everything in between is "?????". >>>>> >>>> >>>> Maybe this is related to the fact that we have function descriptors on >>>> BE and simple function pointers on LE. You may have a look at the >>>> elf-decoder for ppc64 to find some more information. >>> >>> Yes, indeed. With the following patch, the mixed mode option works fine on >>> ppc64 little endian, >>> but not on big endian: >>> >>> Index: >>> jdk9-dev/hotspot/agent/src/share/classes/sun/jvm/hotspot/debugger/linux/ppc64/LinuxPPC64CFrame.java >>> =================================================================== >>> --- >>> jdk9-dev.orig/hotspot/agent/src/share/classes/sun/jvm/hotspot/debugger/linux/ppc64/LinuxPPC64CFrame.java >>> +++ >>> jdk9-dev/hotspot/agent/src/share/classes/sun/jvm/hotspot/debugger/linux/ppc64/LinuxPPC64CFrame.java >>> @@ -60,14 +60,15 @@ final public class LinuxPPC64CFrame exte >>> return null; >>> } >>> >>> - Address nextSP = sp.getAddressAt( PPC64ThreadContext.SP * address_size >>> + PPC64_STACK_BIAS); >>> + Address nextSP = sp.getAddressAt(0); >>> if (nextSP == null) { >>> return null; >>> } >>> - Address nextPC = sp.getAddressAt(PPC64ThreadContext.PC * address_size >>> + PPC64_STACK_BIAS); >>> + Address nextPC = sp.getAddressAt(2 * address_size); >>> if (nextPC == null) { >>> return null; >>> } >>> + >>> return new LinuxPPC64CFrame(dbg, nextSP, nextPC,address_size); >>> } >>> >>> ------------------------------------------------- >>> >>> I see that ppc64 fixups were made in the hotspot utilities (by you) about a >>> year ago >>> (http://cr.openjdk.java.net/~simonis/webrevs/8019929.v3/). We obviously >>> need something >>> similar in the hotspot agent native code that implements the JNI call >>> 'lookupByAddress0'. >>> I hacked the build_symtab_internal() function in >>> hotspot/agent/src/os/linux/symtab.c and >>> see that the symbol "offset" we're getting is really the address of the >>> symbol's opd. >>> I'm not sure where to start to fix this, so if you have any suggestions, >>> I'm all ears. :-) >>> >>> Thanks. >>> -Maynard >>>> >>>>>> >>>>>> Regarding your "test.java" example - how do you use it? >>>>>> >>>>>> If I just attach with jstack to the Java process which runs >>>>>> "test.java" I get the correct stack trace of all threads. But I think >>>>>> that's actual no SA-functionality but a VM-feature (the same that can >>>>>> be triggered by sending kill -SIGQUIT to java process). >>>>>> >>>>>> If I attach with "jstack -F" I see the problems you mentioned. First I >>>>>> didn't saw any frame at all which confused me but then I also saw the >>>>>> two cases mentioned by you. I'll need to have a closer look what >>>>>> happens. >>>>> >>>>> I was just running the 'test' java app and, in another session, killing >>>>> it with SIGSEGV. >>>>> To be honest, I wasn't aware of the 'jstack -F' option. >>>>> >>>> >>>> Another possibility I've just found out is to create a core from gdb >>>> with the 'generate-core-file' command. You can than still inspect the >>>> original program in gdb while debugging how jstack is working on the >>>> core file. >>>> >>>>> -Maynard >>>>> >>>>>> >>>>>> Regards, >>>>>> Volker >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Nov 14, 2014 at 7:09 PM, Maynard Johnson <[email protected]> >>>>>> wrote: >>>>>>> When Hotspot SA tools jmap, jstack, and jsadebugd are run against a >>>>>>> core file, they fail with the following runtime exception: >>>>>>> >>>>>>> OS/CPU combination linux/ppc64 not yet supported >>>>>>> >>>>>>> I will post a patch set that adds this support. The patch set consists >>>>>>> of the following patches: >>>>>>> >>>>>>> PATCH 1/2: Updates to non-Java files to support linux/ppc64 Hotspot SA >>>>>>> with core files >>>>>>> >>>>>>> PATCH 2/2: New PPC64 class files (and updates to generic files) to >>>>>>> support linux/ppc64 Hotspot SA with core files >>>>>>> >>>>>>> These two patches apply cleanly to a November 13 pull of the jdk9-dev >>>>>>> upstream sources. >>>>>>> >>>>>>> ------------ >>>>>>> Open issues: >>>>>>> ------------ >>>>>>> 1) The jstack tool does not print a stack entry for the 'main()' >>>>>>> method of the Java >>>>>>> workload (attached) under test. For example: >>>>>>> >>>>>>> (Note: Addresses and method signatures elided for brevity.) >>>>>>> >>>>>>> Thread 24358: (state = IN_JAVA, current Java SP = null >>>>>>> ) >>>>>>> - java.lang.String.getChars(...) @bci=58, line=814, pc=..., >>>>>>> Method*=... (Compiled frame; ... imprecise) >>>>>>> - test.run_test() @bci=80, line=33, pc=..., Method*=... >>>>>>> (Compiled frame) >>>>>>> ==> (Expect an entry for test.main() here) >>>>>>> >>>>>>> 2) The jstack tool sometimes prints what appears to be two complete >>>>>>> stacks for the Java workload. For example: >>>>>>> >>>>>>> Thread 24779: (state = IN_JAVA, current Java SP = null >>>>>>> ) >>>>>>> - java.lang.String.getChars(...) @bci=58, line=814, pc=..., >>>>>>> Method*=... (Compiled frame; ... imprecise) >>>>>>> - test.run_test() @bci=80, line=33, pc=..., Method*=... >>>>>>> (Compiled frame) >>>>>>> - test.get_my_chars(...) @bci=39, line=15, pc=..., Method*=... >>>>>>> (Compiled frame) >>>>>>> - test.run_test() @bci=92, line=34, pc=..., Method*=... >>>>>>> (Compiled frame) >>>>>>> >>>>>>> Again, the 'test.main' method is missing, but there's also the >>>>>>> anomaly of the >>>>>>> test.run_test' method showing up twice in the stack, implying >>>>>>> that it is called >>>>>>> by 'test.get_my_chars' at line 15. But that that is not >>>>>>> accurate. In fact, run_test >>>>>>> does call String.getChars at line 33 *and* it calls >>>>>>> test.get_my_chars at line 34 -- >>>>>>> but these are totally distinct call graphs. Somehow, we are >>>>>>> seeing these two distinct >>>>>>> stacks in the core file, which seems impossible. >>>>>>> >>>>>>> --------- >>>>>>> >>>>>>> Any help offered on these two open issues would be greatly appreciated. >>>>>>> >>>>>>> -Maynard >>>>>> >>>>> >>>> >>> >
