On 11/03/2020 4:03 pm, Yasumasa Suenaga wrote:
Thanks David!Can you share native backtrace? (Did /opt/core.sh collect it?)
There is a core file but I can't process it, sorry. David -----
Yasumasa On 2020/03/11 14:59, David Holmes wrote:Hi Yasumasa, Partial hs_err info below. David ----- # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x00007fdf2000e87c, pid=29798, tid=29800 ## JRE version: Java(TM) SE Runtime Environment (15.0) (fastdebug build 15-internal+0-2020-03-11-0447267.suenaga.source) # Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 15-internal+0-2020-03-11-0447267.suenaga.source, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64)# Problematic frame:# C [libsaproc.so+0x487c] DwarfParser::process_dwarf(unsigned long)+0x2c## Core dump will be written. Default location: Core dumps may be processed with "/opt/core.sh %p" (or dumping to /opt/mach5/mesos/work_dir/slaves/90726e33-be99-4e27-9d68-25dad266ef13-S5982/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/5f116aa5-2fc4-44e1-8808-1284111132ed/runs/908d4a0c-e2df-4273-bb74-b899888c3b6a/testoutput/test-support/jtreg_open_test_hotspot_jtreg_tier1_serviceability/scratch/0/core.29798)# # If you would like to submit a bug report, please visit: # https://bugreport.java.com/bugreport/crash.jsp # The crash happened outside the Java Virtual Machine in native code. # See problematic frame for where to report the bug. # --------------- S U M M A R Y ------------Command Line: -Denv.class.path=/opt/mach5/mesos/work_dir/slaves/90726e33-be99-4e27-9d68-25dad266ef13-S5982/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/5f116aa5-2fc4-44e1-8808-1284111132ed/runs/908d4a0c-e2df-4273-bb74-b899888c3b6a/testoutput/test-support/jtreg_open_test_hotspot_jtreg_tier1_serviceability/classes/0/serviceability/sa/TestJhsdbJstackMixed.d:/opt/mach5/mesos/work_dir/jib-master/install/2020-03-11-0447267.suenaga.source/src.full/open/test/hotspot/jtreg/serviceability/sa:/opt/mach5/mesos/work_dir/slaves/90726e33-be99-4e27-9d68-25dad266ef13-S5982/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/5f116aa5-2fc4-44e1-8808-1284111132ed/runs/908d4a0c-e2df-4273-bb74-b899888c3b6a/testoutput/test-support/jtreg_open_test_hotspot_jtreg_tier1_serviceability/classes/0/test/lib:/opt/mach5/mesos/work_dir/jib-master/install/2020-03-11-0447267.suenaga.source/src.full/open/test/lib:/opt/mach5/mesos/work_dir/jib-master/install/jtreg/5.0/b01/bundles/jtreg_bin-5.0.zip/jtreg/lib/javatest.jar:/opt/mach5/mesos/work_dir/jib-master/install/jtreg/5.0/b01/bundles/jtreg_bin-5.0.zip/jtreg/lib/jtreg.jar -Dapplication.home=/opt/mach5/mesos/work_dir/jib-master/install/2020-03-11-0447267.suenaga.source/linux-x64-debug.jdk/jdk-15/fastdebug -Xms8m -Djdk.module.main=jdk.hotspot.agent jdk.hotspot.agent/sun.jvm.hotspot.SALauncher jstack --mixed --pid 29770Time: Wed Mar 11 05:20:57 2020 UTC elapsed time: 3.927809 seconds (0d 0h 0m 3s)--------------- T H R E A D ---------------Current thread (0x00007fdf5c032000): JavaThread "main" [_thread_in_native, id=29800, stack(0x00007fdf63a9e000,0x00007fdf63b9f000)]Stack: [0x00007fdf63a9e000,0x00007fdf63b9f000], sp=0x00007fdf63b9d190, free space=1020k Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)C [libsaproc.so+0x487c] DwarfParser::process_dwarf(unsigned long)+0x2cj sun.jvm.hotspot.debugger.linux.amd64.DwarfParser.processDwarf0(J)V+0 jdk.hotspot.agent@15-internal j sun.jvm.hotspot.debugger.linux.amd64.DwarfParser.processDwarf(Lsun/jvm/hotspot/debugger/Address;)V+7 jdk.hotspot.agent@15-internal j sun.jvm.hotspot.debugger.linux.amd64.LinuxAMD64CFrame.getTopFrame(Lsun/jvm/hotspot/debugger/linux/LinuxDebugger;Lsun/jvm/hotspot/debugger/Address;Lsun/jvm/hotspot/debugger/ThreadContext;)Lsun/jvm/hotspot/debugger/linux/amd64/LinuxAMD64CFrame;+38 jdk.hotspot.agent@15-internal j sun.jvm.hotspot.debugger.linux.LinuxCDebugger.topFrameForThread(Lsun/jvm/hotspot/debugger/ThreadProxy;)Lsun/jvm/hotspot/debugger/cdbg/CFrame;+116 jdk.hotspot.agent@15-internal j sun.jvm.hotspot.tools.PStack.run(Ljava/io/PrintStream;Lsun/jvm/hotspot/debugger/Debugger;)V+125 jdk.hotspot.agent@15-internal j sun.jvm.hotspot.tools.PStack.run(Ljava/io/PrintStream;)V+11 jdk.hotspot.agent@15-internalj sun.jvm.hotspot.tools.PStack.run()V+4 jdk.hotspot.agent@15-internal j sun.jvm.hotspot.tools.JStack.run()V+55 jdk.hotspot.agent@15-internalj sun.jvm.hotspot.tools.Tool.startInternal()V+87 jdk.hotspot.agent@15-internal j sun.jvm.hotspot.tools.Tool.start([Ljava/lang/String;)I+359 jdk.hotspot.agent@15-internal j sun.jvm.hotspot.tools.Tool.execute([Ljava/lang/String;)V+4 jdk.hotspot.agent@15-internal j sun.jvm.hotspot.tools.JStack.runWithArgs([Ljava/lang/String;)V+99 jdk.hotspot.agent@15-internal j sun.jvm.hotspot.SALauncher.runJSTACK([Ljava/lang/String;)V+58 jdk.hotspot.agent@15-internal j sun.jvm.hotspot.SALauncher$$Lambda$3.accept(Ljava/lang/Object;)V+4 jdk.hotspot.agent@15-internal j sun.jvm.hotspot.SALauncher.main([Ljava/lang/String;)V+158 jdk.hotspot.agent@15-internalv ~StubRoutines::call_stubV [libjvm.so+0xc2291c] JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0x6ac V [libjvm.so+0xd31970] jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*) [clone .isra.140] [clone .constprop.263]+0x370V [libjvm.so+0xd36202] jni_CallStaticVoidMethod+0x222 C [libjli.so+0x4bed] JavaMain+0xbcd C [libjli.so+0x80a9] ThreadJavaMain+0x9 Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)j sun.jvm.hotspot.debugger.linux.amd64.DwarfParser.processDwarf0(J)V+0 jdk.hotspot.agent@15-internal j sun.jvm.hotspot.debugger.linux.amd64.DwarfParser.processDwarf(Lsun/jvm/hotspot/debugger/Address;)V+7 jdk.hotspot.agent@15-internal j sun.jvm.hotspot.debugger.linux.amd64.LinuxAMD64CFrame.getTopFrame(Lsun/jvm/hotspot/debugger/linux/LinuxDebugger;Lsun/jvm/hotspot/debugger/Address;Lsun/jvm/hotspot/debugger/ThreadContext;)Lsun/jvm/hotspot/debugger/linux/amd64/LinuxAMD64CFrame;+38 jdk.hotspot.agent@15-internal j sun.jvm.hotspot.debugger.linux.LinuxCDebugger.topFrameForThread(Lsun/jvm/hotspot/debugger/ThreadProxy;)Lsun/jvm/hotspot/debugger/cdbg/CFrame;+116 jdk.hotspot.agent@15-internal j sun.jvm.hotspot.tools.PStack.run(Ljava/io/PrintStream;Lsun/jvm/hotspot/debugger/Debugger;)V+125 jdk.hotspot.agent@15-internal j sun.jvm.hotspot.tools.PStack.run(Ljava/io/PrintStream;)V+11 jdk.hotspot.agent@15-internalj sun.jvm.hotspot.tools.PStack.run()V+4 jdk.hotspot.agent@15-internal j sun.jvm.hotspot.tools.JStack.run()V+55 jdk.hotspot.agent@15-internalj sun.jvm.hotspot.tools.Tool.startInternal()V+87 jdk.hotspot.agent@15-internal j sun.jvm.hotspot.tools.Tool.start([Ljava/lang/String;)I+359 jdk.hotspot.agent@15-internal j sun.jvm.hotspot.tools.Tool.execute([Ljava/lang/String;)V+4 jdk.hotspot.agent@15-internal j sun.jvm.hotspot.tools.JStack.runWithArgs([Ljava/lang/String;)V+99 jdk.hotspot.agent@15-internal j sun.jvm.hotspot.SALauncher.runJSTACK([Ljava/lang/String;)V+58 jdk.hotspot.agent@15-internal j sun.jvm.hotspot.SALauncher$$Lambda$3.accept(Ljava/lang/Object;)V+4 jdk.hotspot.agent@15-internal j sun.jvm.hotspot.SALauncher.main([Ljava/lang/String;)V+158 jdk.hotspot.agent@15-internalv ~StubRoutines::call_stubsiginfo: si_signo: 11 (SIGSEGV), si_code: 2 (SEGV_ACCERR), si_addr: 0x00007fded5076b79Register to memory mapping: RAX=0x00007f7e4dfe3229 is an unknown valueRBX=0x00007fdf5c4d7080 points into unknown readable memory: 80 23 07 d4 de 7f 00 00 RCX=0x00007fded4072380 points into unknown readable memory: 2f 75 73 72 2f 6c 69 62RDX=0x00007fded4076b85 points into unknown readable memory: 01 00 00RSP=0x00007fdf63b9d190 is pointing into the stack for thread: 0x00007fdf5c032000 RBP=0x00007fdf63b9d1b0 is pointing into the stack for thread: 0x00007fdf5c032000RSI=0x0000000000000004 is an unknown valueRDI=0x00007fdf5c4d7080 points into unknown readable memory: 80 23 07 d4 de 7f 00 00 R8 =0x000000000146c380 points into unknown readable memory: 02 00 00 00 00 00 00 00 R9 =0x00007fded4076b79 points into unknown readable memory: 7a 52 00 01 78 10 01R10=0x00000000ffffffff is an unknown value R11=0x000000000100527a is an unknown value R12=0x00007fded5076b79 is an unknown value R13=0x00007f7da2f8e68a is an unknown value R14=0x00007f7dbdf62b1d is an unknown value R15=0x00007fdf5c032000 is a thread Registers:RAX=0x00007f7e4dfe3229, RBX=0x00007fdf5c4d7080, RCX=0x00007fded4072380, RDX=0x00007fded4076b85 RSP=0x00007fdf63b9d190, RBP=0x00007fdf63b9d1b0, RSI=0x0000000000000004, RDI=0x00007fdf5c4d7080 R8 =0x000000000146c380, R9 =0x00007fded4076b79, R10=0x00000000ffffffff, R11=0x000000000100527a R12=0x00007fded5076b79, R13=0x00007f7da2f8e68a, R14=0x00007f7dbdf62b1d, R15=0x00007fdf5c032000 RIP=0x00007fdf2000e87c, EFLAGS=0x0000000000010206, CSGSFS=0x002b000000000033, ERR=0x0000000000000004TRAPNO=0x000000000000000e Top of Stack: (sp=0x00007fdf63b9d190) 0x00007fdf63b9d190: 00007fdf209d0980 0000000000000000 0x00007fdf63b9d1a0: 00007fdf209d0980 00007fdf63b9d258 0x00007fdf63b9d1b0: 00007fdf63b9d228 00007fdf44778dbe 0x00007fdf63b9d1c0: 000000000146c380 00007fdf5c032000 Instructions: (pc=0x00007fdf2000e87c) 0x00007fdf2000e77c: 89 43 18 4d 85 f6 75 0f eb 2a 66 2e 0f 1f 84 00 0x00007fdf2000e78c: 00 00 00 00 48 89 c2 48 8d 42 01 48 89 43 08 80 0x00007fdf2000e79c: 78 ff 00 78 ef 48 8d 42 02 48 89 43 08 0f b6 42 0x00007fdf2000e7ac: 01 88 43 10 48 c7 43 28 00 00 00 00 4c 89 e1 48 0x00007fdf2000e7bc: 89 df 31 f6 48 b8 07 00 00 00 10 00 00 00 c6 43 0x00007fdf2000e7cc: 3c 00 48 c7 c2 ff ff ff ff 48 89 43 14 48 c7 43 0x00007fdf2000e7dc: 30 00 00 00 00 c7 43 38 00 00 00 00 e8 13 fb ff 0x00007fdf2000e7ec: ff 4c 89 6b 08 48 83 c4 18 5b 41 5c 41 5d 41 5e 0x00007fdf2000e7fc: 41 5f 5d c3 83 e7 40 0f 84 63 ff ff ff 48 c7 c2 0x00007fdf2000e80c: ff ff ff ff 48 d3 e2 49 09 d0 e9 51 ff ff ff 90 0x00007fdf2000e81c: 0f 1f 40 00 0f b6 47 10 83 e0 07 3c 02 74 0a 76 0x00007fdf2000e82c: 1b 3c 03 74 04 3c 04 75 17 48 8b 57 08 8b 02 48 0x00007fdf2000e83c: 83 c2 04 48 89 57 08 c3 0f 1f 40 00 84 c0 74 e9 0x00007fdf2000e84c: 31 c0 c3 90 55 41 ba ff ff ff ff 48 89 e5 41 56 0x00007fdf2000e85c: 41 55 49 89 f5 41 54 53 48 8b 07 48 89 fb 4c 8b 0x00007fdf2000e86c: a0 28 11 00 00 eb 09 0f 1f 44 00 00 4c 89 63 08 0x00007fdf2000e87c: 41 8b 04 24 4d 8d 4c 24 04 4c 89 4b 08 4c 39 d0 0x00007fdf2000e88c: 75 0a 49 8b 44 24 04 4d 8d 4c 24 0c 45 8b 19 4d 0x00007fdf2000e89c: 8d 24 01 49 8d 41 04 48 89 43 08 45 85 db 74 cc 0x00007fdf2000e8ac: 48 89 df e8 8c f9 ff ff 48 8b 13 41 89 c6 4c 03 0x00007fdf2000e8bc: b2 18 11 00 00 e8 5a ff ff ff 89 c0 4c 01 f0 4c 0x00007fdf2000e8cc: 39 e8 76 a8 4d 39 ee 77 a3 44 89 da 4c 89 ce e8 0x00007fdf2000e8dc: 90 fd ff ff 48 8b 43 08 31 c9 31 ff 48 83 c0 01 0x00007fdf2000e8ec: 0f 1f 40 00 48 89 43 08 0f b6 70 ff 49 89 c0 48 0x00007fdf2000e8fc: 83 c0 01 48 89 f2 83 e2 7f 48 d3 e2 83 c1 07 48 0x00007fdf2000e90c: 09 d7 40 84 f6 78 dd 4c 01 c7 4c 89 e1 4c 89 ea 0x00007fdf2000e91c: 4c 89 f6 48 89 7b 08 48 89 df e8 d5 f9 ff ff 5b 0x00007fdf2000e92c: 31 c0 41 5c 41 5d 41 5e 5d c3 66 2e 0f 1f 84 00 0x00007fdf2000e93c: 00 00 00 00 55 48 89 e5 41 54 53 48 81 ec d0 00 0x00007fdf2000e94c: 00 00 48 89 b5 48 ff ff ff 48 89 95 50 ff ff ff 0x00007fdf2000e95c: 48 89 8d 58 ff ff ff 4c 89 85 60 ff ff ff 4c 89 0x00007fdf2000e96c: 8d 68 ff ff ff 84 c0 74 23 0f 29 85 70 ff ff ff On 11/03/2020 3:52 pm, Yasumasa Suenaga wrote:Hi Kevin,I saw 2 errors on submit repo (mach5-one-ysuenaga-JDK-8234624-5-20200311-0209-9358475). So I tweaked my patch, but I saw the crash again (mach5-one-ysuenaga-JDK-8234624-5-20200311-0448-9361448).Last change on submit repo is here: http://cr.openjdk.java.net/~ysuenaga/JDK-8234624/webrev.05-2/ Can you share details on submit repo? Thanks, Yasumasa On 2020/03/11 11:07, Yasumasa Suenaga wrote:Hi Kevin,I guess first program header in the libraries which are on your machine has exec flag (you can check it with `readelf -l`). So I tweaked my patch (initial value of exec_start and exec_end set to -1) in new webrev.http://cr.openjdk.java.net/~ysuenaga/JDK-8234624/webrev.05/This webrev contains the fix for your comment (typo and DW_CFA_advance_loc4).Thanks, Yasumasa On 2020/03/11 8:53, Kevin Walls wrote:Hi - In testing I wasn't seeing any of the Dwarf code triggered.With LIBSAPROC_DEBUG set I'm getting the "Could not find executable section in" for lots of / maybe all the libraries...src/jdk.hotspot.agent/linux/native/libsaproc/libproc_impl.c if (fill_instr_info(newlib)) { if (!read_eh_frame(ph, newlib)) { fill_instr_info is failing, and we never get to read_eh_frame(). output like: libsaproc DEBUG: [0] vaddr = 0x0, memsz = 0xaba4, filesz = 0xaba4libsaproc DEBUG: Could not find executable section in /lib/x86_64-linux-gnu/libnss_nis-2.27.so(similar for all libraries). fill_instr fails if: if ((lib->exec_start == 0L) || (lib->exec_end == 0L))...but isn't exec_start relative to the library address? It's the value of ph->vaddr and it is often zero.I added some booleans and did:185 if ((lib->exec_start == 0L) || (lib->exec_start > ph->p_vaddr)) {186 lib->exec_start = ph->p_vaddr; 187 found_start =true; 188 } (similarly for end) and only failed if: 201 if (!found_start || !found_end) { 202 return false; ...and now it's better. I go from: ----------------- 3306 ----------------- 0x00007f75824acd2d __GI___pthread_timedjoin_ex + 0x17d to: ----------------- 31127 ----------------- 0x00007fa284d78d2d __GI___pthread_timedjoin_ex + 0x17d 0x00007fa2857aaf2d CallJavaMainInNewThread + 0xad 0x00007fa2857a74ed ContinueInNewThread + 0x4d 0x00007fa2857a8c49 JLI_Launch + 0x1529 0x000055af1b78db1c main + 0x11c Thanks Kevin On 10/03/2020 12:36, Yasumasa Suenaga wrote:Hi Kevin, Thanks for your comment! On 2020/03/10 18:58, Kevin Walls wrote:Hi Yasumasa ,The changes build OK for me in the latest jdk, and things still work. I have not yet seen the dwarf usage in action: I've tried a couple of different systems and so far have not reproduced the problem, i.e. jstack has not failed on native frames.I may need more recent basic libraries, will look again for somewhere where the problem happens and get back to you as I really want to run the changes.You can see the problem with JShell. Some Java frames would not be seen in mixed jstack.I have mostly minor other comments which don't need a new webrev, some just comments for the future:src/jdk.hotspot.agent/linux/native/libsaproc/dwarf.cpp: DW_CFA_nop - shouldn't this continue instead of return?(It may "never" happen, but a nop could appear within some other instructions?)DW_CFA_nop is used for padding, so we can ignore (return immediately) it.DW_CFA_remember_state: a minor typo in the comment, "DW_CFA_remenber_state".I will fix it.We handle DW_CFA_advance_loc, and _loc1 and _loc2, but not DW_CFA_advance_loc4. I thought that was odd, but maybe addresses in these tables never increase by 4-byte amounts, would this mean a lot of code on one line. 8-) So maybe it's never used in practice, if you think it's unnecessary no problem, maybe a comment, or add it for robustness.I will add DW_CFA_advance_loc4.General-purpose methods like read_leb128(), get_entry_length(), get_decoded_value() specifically update the _buf pointer in this DwarfParser.DwarfParser::process_dwarf() moves _buf.It calls process_cie() which reads, moves _buf and restores it to the original position, then we read augmentation_length from where _buf is. I'm not sure if that's wrong, or if I just need to read again about the CIE/etc layout.I don't really want to suggest making the code pass around a current _buf for the invocation of these general purpose methods, but just wanted to comment that if these get used more widely that might become necessary.I saw GDB and binutils source for creating this patch.They seems to process similar code because we need to calculate DWARF instructions one-by-one to get the value which relates to specified PC.Similarly in future, if this DWARF support code became used more widely, it might want to move to anOS-neutral directory? It's odd to label it as Linux-specific.Windows does not use DWARF at least, it uses another feature.https://urldefense.com/v3/__https://docs.microsoft.com/en-us/cpp/build/exception-handling-x64?view=vs-2019__;!!GqivPVa7Brio!J801oKj34Q7f-4SzAWGKL67e6Xq2yMlV6f01eqp_fqqhqgKktCBiUi2RUaTtALtpRg$I'm not sure other platforms (Solaris, macOS) uses DWARF.If DWARF is used in them, I can move DWARF related code to posix directory.src/jdk.hotspot.agent/linux/native/libsaproc/dwarf.hpp:Thanks for changing "can_parsable" which was in the earlier version. 8-)These are just comments to mainly say it looks good, and somebody else out there has read it. I will look for a system that shows the problem, and get back to you again!Thanks, YasumasaMany thanks Kevin On 27/02/2020 05:13, Yasumasa Suenaga wrote:Hi all,webrev.03 cannot be applied to current jdk/jdk due to 8239224 and 8239462 changes (they updated copyright year). So I modified webrev (only copyright year changes) to be able to apply to current jdk/jdk.Could you review it? http://cr.openjdk.java.net/~ysuenaga/JDK-8234624/webrev.04/ I need one more reviewer to push. Thanks, Yasumasa On 2020/02/17 13:07, Yasumasa Suenaga wrote:PING: Could you review it?JBS: https://bugs.openjdk.java.net/browse/JDK-8234624webrev: http://cr.openjdk.java.net/~ysuenaga/JDK-8234624/webrev.03/This change has been already reviewed by Serguei. I need one more reviewer to push. Thanks, Yasumasa On 2020/02/03 1:37, Yasumasa Suenaga wrote:PING: Could you reveiw this change?JBS: https://bugs.openjdk.java.net/browse/JDK-8234624webrev: http://cr.openjdk.java.net/~ysuenaga/JDK-8234624/webrev.03/I believe this change helps troubleshooter to fight to postmortem analysis.Thanks, Yasumasa On 2020/01/19 3:16, Yasumasa Suenaga wrote:PING: Could you review it? JBS: https://bugs.openjdk.java.net/browse/JDK-8234624webrev: http://cr.openjdk.java.net/~ysuenaga/JDK-8234624/webrev.03/I updated webrev. I discussed with Serguei in off list, and I refactored webrev.02 . It has passed tests on submit repo (mach5-one-ysuenaga-JDK-8234624-4-20200118-1353-8149549).Thanks, Yasumasa On 2019/12/15 10:51, Yasumasa Suenaga wrote:Hi Serguei, Thanks for your comment! I refactored LinuxCDebugger and LinuxAMD64CFrame in new webrev.Also I fixed to free lib->eh_frame.data in libproc_impl.c as Dmitry said.http://cr.openjdk.java.net/~ysuenaga/JDK-8234624/webrev.02/This change has been passed all tests on submit repo (mach5-one-ysuenaga-JDK-8234624-3-20191214-1527-7538487).Thanks, Yasumasa On 2019/12/14 10:02, serguei.spit...@oracle.com wrote:Hi Yasumasa, This is nice move in general. Thank you for working on this!http://cr.openjdk.java.net/~ysuenaga/JDK-8234624/webrev.01/src/jdk.hotspot.agent/share/classes/sun/jvm/hotspot/debugger/linux/LinuxCDebugger.java.frames.html96 long libptr = dbg.findLibPtrByAddress(pc); 97 if (libptr == 0L) { // Java frame 98 Address rbp = context.getRegisterAsAddress(AMD64ThreadContext.RBP); 99 if (rbp == null) { 100 return null; 101 } 102 return new LinuxAMD64CFrame(dbg, rbp, pc, null); 103 } else { // Native frame 104 DwarfParser dwarf; 105 try { 106 dwarf = new DwarfParser(libptr); 107 } catch (DebuggerException e) { 108 Address rbp = context.getRegisterAsAddress(AMD64ThreadContext.RBP); 109 if (rbp == null) { 110 return null; 111 } 112 return new LinuxAMD64CFrame(dbg, rbp, pc, null); 113 } 114 dwarf.processDwarf(pc); 115 Address cfa = ((dwarf.getCFARegister() == AMD64ThreadContext.RBP) && 116 !dwarf.isBPOffsetAvailable()) 117 ? context.getRegisterAsAddress(AMD64ThreadContext.RBP) 118 : context.getRegisterAsAddress(dwarf.getCFARegister()) 119 .addOffsetTo(dwarf.getCFAOffset()); 120 if (cfa == null) { 121 return null; 122 } 123 return new LinuxAMD64CFrame(dbg, cfa, pc, dwarf); 124 }I'd suggest to simplify the logic by refactoring to something like below:long libptr = dbg.findLibPtrByAddress(pc);Address cfa = context.getRegisterAsAddress(AMD64ThreadContext.RBP); // Java frameDwarfParser dwarf = null; if (libptr != 0L) { // Native frame try { dwarf = new DwarfParser(libptr); dwarf.processDwarf(pc);Address cfa = ((dwarf.getCFARegister() == AMD64ThreadContext.RBP) &&!dwarf.isBPOffsetAvailable())? context.getRegisterAsAddress(AMD64ThreadContext.RBP) : context.getRegisterAsAddress(dwarf.getCFARegister()).addOffsetTo(dwarf.getCFAOffset());} catch (DebuggerException e) { // bail out to Java frame case} } if (cfa == null) { return null; } return new LinuxAMD64CFrame(dbg, cfa, pc, dwarf);http://cr.openjdk.java.net/~ysuenaga/JDK-8234624/webrev.01/src/jdk.hotspot.agent/share/classes/sun/jvm/hotspot/debugger/linux/amd64/LinuxAMD64CFrame.java.frames.html58 long ofs = useDwarf ? dwarf.getReturnAddressOffsetFromCFA() Better to rename 'ofs' => 'offs'.77 nextCFA = nextCFA.addOffsetTo(- nextDwarf.getBasePointerOffsetFromCFA());Extra space after '-' sign.71 private Address getNextCFA(DwarfParser nextDwarf, ThreadContext context) {It feels like the logic has to be somehow refactored/simplified as several typical fragments appears in slightly different contexts.But it is not easy to understand what it is.Could you, please, add some comments to key places explaining this logic. Then I'll check if it is possible to make it a little bit simpler.109 private CFrame javaSender(ThreadContext context) { 110 Address nextCFA; 111 Address nextPC; 112 113 nextPC = getNextPC(false); 114 if (nextPC == null) { 115 return null; 116 } 117 118 DwarfParser nextDwarf = null; 119 long libptr = dbg.findLibPtrByAddress(nextPC); 120 if (libptr != 0L) { // Native frame 121 try { 122 nextDwarf = new DwarfParser(libptr); 123 } catch (DebuggerException e) { 124 nextCFA = getNextCFA(null, context); 125 return (nextCFA == null) ? null : new LinuxAMD64CFrame(dbg, nextCFA, nextPC, null); 126 } 127 nextDwarf.processDwarf(nextPC); 128 } 129 130 nextCFA = getNextCFA(nextDwarf, context); 131 return (nextCFA == null) ? null 132 : new LinuxAMD64CFrame(dbg, nextCFA, nextPC, nextDwarf); 133 }The above can be simplified if a DebuggerException can not be thrown from processDwarf(nextPC):private CFrame javaSender(ThreadContext context) { Address nextPC = getNextPC(false); if (nextPC == null) { return null; } long libptr = dbg.findLibPtrByAddress(nextPC); DwarfParser nextDwarf = null; if (libptr != 0L) { // Native frame try { nextDwarf = new DwarfParser(libptr); nextDwarf.processDwarf(nextPC);} catch (DebuggerException e) { // Bail out to Java frame} } Address nextCFA = getNextCFA(nextDwarf, context);return (nextCFA == null) ? null : new LinuxAMD64CFrame(dbg, nextCFA, nextPC, nextDwarf);}135 public CFrame sender(ThreadProxy thread) { 136 ThreadContext context = thread.getContext(); 137 138 if (dwarf == null) { // Java frame 139 return javaSender(context); 140 } 141 142 Address nextPC = getNextPC(true); 143 if (nextPC == null) { 144 return null; 145 } 146 147 Address nextCFA; 148 DwarfParser nextDwarf = dwarf; 149 if (!dwarf.isIn(nextPC)) { 150 long libptr = dbg.findLibPtrByAddress(nextPC); 151 if (libptr == 0L) { 152 // Next frame might be Java frame 153 nextCFA = getNextCFA(null, context); 154 return (nextCFA == null) ? null : new LinuxAMD64CFrame(dbg, nextCFA, nextPC, null); 155 } 156 try { 157 nextDwarf = new DwarfParser(libptr); 158 } catch (DebuggerException e) { 159 nextCFA = getNextCFA(null, context); 160 return (nextCFA == null) ? null : new LinuxAMD64CFrame(dbg, nextCFA, nextPC, null); 161 } 162 } 163 164 nextDwarf.processDwarf(nextPC); 165 nextCFA = getNextCFA(nextDwarf, context); 166 return (nextCFA == null) ? null : new LinuxAMD64CFrame(dbg, nextCFA, nextPC, nextDwarf); 167 }This one can be also simplified a little: public CFrame sender(ThreadProxy thread) { ThreadContext context = thread.getContext(); if (dwarf == null) { // Java frame return javaSender(context); } Address nextPC = getNextPC(true); if (nextPC == null) { return null; } DwarfParser nextDwarf = null; if (!dwarf.isIn(nextPC)) { long libptr = dbg.findLibPtrByAddress(nextPC); if (libptr != 0L) { try { nextDwarf = new DwarfParser(libptr); nextDwarf.processDwarf(nextPC);} catch (DebuggerException e) { // Bail out to Java frame} } } Address nextCFA = getNextCFA(nextDwarf, context);return (nextCFA == null) ? null : new LinuxAMD64CFrame(dbg, nextCFA, nextPC, nextDwarf);} Finally, it looks like just one method could replace bothsender(ThreadProxy thread) and javaSender(ThreadContext context):private CFrame commonSender(ThreadProxy thread) { ThreadContext context = thread.getContext(); Address nextPC = getNextPC(false); if (nextPC == null) { return null; } DwarfParser nextDwarf = null; long libptr = dbg.findLibPtrByAddress(nextPC); if (dwarf == null || !dwarf.isIn(nextPC)) { long libptr = dbg.findLibPtrByAddress(nextPC); if (libptr != 0L) { try { nextDwarf = new DwarfParser(libptr); nextDwarf.processDwarf(nextPC);} catch (DebuggerException e) { // Bail out to Java frame} } } Address nextCFA = getNextCFA(nextDwarf, context);return (nextCFA == null) ? null : new LinuxAMD64CFrame(dbg, nextCFA, nextPC, nextDwarf);} I'm still reviewing the dwarf parser files. Thanks, Serguei On 11/28/19 4:39 AM, Yasumasa Suenaga wrote:Hi,I refactored LinuxAMD64CFrame.java . It works fine in serviceability/sa tests and all tests on submit repo (mach5-one-ysuenaga-JDK-8234624-2-20191128-0928-7059923).Could you review new webrev? http://cr.openjdk.java.net/~ysuenaga/JDK-8234624/webrev.01/ The diff from previous webrev is here: http://hg.openjdk.java.net/jdk/submit/rev/4bc47efbc90b Thanks, Yasumasa On 2019/11/25 14:08, Yasumasa Suenaga wrote:Hi all, Please review this change: JBS: https://bugs.openjdk.java.net/browse/JDK-8234624webrev: http://cr.openjdk.java.net/~ysuenaga/JDK-8234624/webrev.00/According to 2.7 Stack Unwind Algorithm in System V Application Binary Interface AMD64 Architecture Processor Supplement [1], we need to use DWARF in .eh_frame or .debug_framefor stack unwinding.As JDK-8022183 said, omit-frame-pointer is enabled by default since GCC 4.6, so systemlibrary (e.g. libc) might be compiled with this feature.However `jhsdb jstack --mixed` does not do so, it uses base pointer register (RBP).So it might be lack of stack frames. I guess JDK-8219201 is caused by same issue. Thanks, Yasumasa[1] https://urldefense.com/v3/__https://software.intel.com/sites/default/files/article/402129/mpx-linux64-abi.pdf__;!!GqivPVa7Brio!J801oKj34Q7f-4SzAWGKL67e6Xq2yMlV6f01eqp_fqqhqgKktCBiUi2RUaQusmjOqA$
