Re: dtrace ustack kernel panic
on 03/08/2011 01:39 maestro something said the following: hm userland and CTF not that I know. I only recompiled the kernel with CTF. Do I have to recompile userland (and would that be make world?) If so can i just pass the same WITH_CTF=1 option to make? I _think_ (as in: not sure) that for ustack() to work correctly it needs CTF information for userland bits. I've never done this myself, so be warned, but I guess that, yes, you need to recompile/reinstall world with WITH_CTF=1. Also, I seem to recall that there were some reports that some things were getting broken because of WITH_CTF=1, so be cautious, plan a way to recover your system from non-working world. -- Andriy Gapon ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: dtrace ustack kernel panic
on 03/08/2011 01:46 maestro something said the following: What do I have to recompile for the patch to be picked up? make libproc in /usr/src/lib Correct way is: $ cd /usr/src/lib/libproc $ make obj $ make depend $ make $ make install -- Andriy Gapon ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: dtrace ustack kernel panic
on 03/08/2011 03:47 maestro something said the following: now it seems that the ustack() action on i386 and FreeBSD-9.0BETA1 almost works. Almost because dtrace still exits (this time without an error message) The process for which the ustack should be genereated is terminated Here are the specifics: fb90i386# dtrace -n 'syscall::accept:return / execname == nc / { ustack();}' dtrace: description 'syscall::accept:return ' matched 1 probe CPU IDFUNCTION:NAME 0 46457accept:return libc.so.7`__sys_accept+0x7 0x3 0xed96e824 This alsmost looks like a stack trace I'd say However, dtrace quits once I connect to nc nc quits with the following error message fb90i386# nc -vl nc: Polling Error: Interrupted system call This could be a combination of several factors. First, for some reason nc gets EINTR in one of its system calls (poll(2)). Second, nc treats EINTR as a serious error and exits. Third, this is what happens on the dtrace side: 30495 100974 dtrace CALL write(0x1,0x8035c2000,0x1a) 30495 100974 dtrace GIO fd 1 wrote 26 bytes libc.so.7`__sys_accept+0xc 30495 100974 dtrace RET write 26/0x1a 30495 100974 dtrace CALL write(0x1,0x8035c2000,0x1) 30495 100974 dtrace GIO fd 1 wrote 1 byte 30495 100974 dtrace RET write 1 30495 100974 dtrace CALL thr_kill(0x1c129,SIGUSR1) 30495 100974 dtrace RET thr_kill 0 30495 100974 dtrace CALL ptrace(PT_IO,0x771e,0x7fffbb10,0) 30495 100974 dtrace RET ptrace 0 30495 100974 dtrace CALL ptrace(PT_IO,0x771e,0x7fffbb10,0) 30495 114985 dtrace RET _umtx_op -1 errno 4 Interrupted system call 30495 100974 dtrace RET ptrace 0 30495 114985 dtrace PSIG SIGUSR1 caught handler=0x80085d610 mask=0x5ffefedf code=0x10007 30495 100974 dtrace CALL _umtx_op(0x802007668,0x15,0x1,0,0) 30495 114985 dtrace CALL sigprocmask(SIG_SETMASK,0x7fbfd9fc,0) 30495 100974 dtrace RET _umtx_op 0 30495 114985 dtrace RET sigprocmask 0 30495 100974 dtrace CALL _umtx_op(0x800a71778,0xf,0,0,0) 30495 114985 dtrace CALL sigreturn(0x7fbfd630) 30495 114985 dtrace RET sigreturn JUSTRETURN 30495 114985 dtrace CALL ptrace(PT_CONTINUE,0x771e,0x1,0) 30495 114985 dtrace RET ptrace 0 30495 114985 dtrace CALL ptrace(PT_IO,0x771e,0x7fbfdee0,0) 30495 114985 dtrace RET ptrace -1 errno 16 Device busy 30495 114985 dtrace CALL _umtx_op(0x80200d668,0x15,0x1,0,0) 30495 114985 dtrace RET _umtx_op 0 30495 114985 dtrace CALL madvise(0x803c52000,0x11000,MADV_FREE) 30495 114985 dtrace RET madvise 0 30495 114985 dtrace CALL madvise(0x803c1d000,0x14000,MADV_FREE) 30495 114985 dtrace RET madvise 0 30495 114985 dtrace CALL madvise(0x803c07000,0x1000,MADV_FREE) 30495 114985 dtrace RET madvise 0 30495 114985 dtrace CALL madvise(0x8037fc000,0x2000,MADV_FREE) 30495 114985 dtrace RET madvise 0 30495 114985 dtrace CALL madvise(0x8037f2000,0x8000,MADV_FREE) 30495 114985 dtrace RET madvise 0 30495 114985 dtrace CALL madvise(0x8037f,0x1000,MADV_FREE) 30495 100974 dtrace RET _umtx_op 0 30495 114985 dtrace RET madvise 0 30495 100974 dtrace CALL ptrace(PT_DETACH,0x771e,0,0) 30495 114985 dtrace CALL madvise(0x8037e,0x9000,MADV_FREE) 30495 100974 dtrace RET ptrace -1 errno 3 No such process Looks like something fishy happens when that SIGUSR1 is generated. I wish a dtrace-for-userland expert and/or a ptrace(2) expert could help us here. -- Andriy Gapon ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: dtrace ustack kernel panic
on 03/08/2011 11:42 Andriy Gapon said the following: on 03/08/2011 03:47 maestro something said the following: now it seems that the ustack() action on i386 and FreeBSD-9.0BETA1 almost works. Almost because dtrace still exits (this time without an error message) The process for which the ustack should be genereated is terminated Here are the specifics: fb90i386# dtrace -n 'syscall::accept:return / execname == nc / { ustack();}' dtrace: description 'syscall::accept:return ' matched 1 probe CPU IDFUNCTION:NAME 0 46457accept:return libc.so.7`__sys_accept+0x7 0x3 0xed96e824 This alsmost looks like a stack trace I'd say However, dtrace quits once I connect to nc nc quits with the following error message fb90i386# nc -vl nc: Polling Error: Interrupted system call This could be a combination of several factors. First, for some reason nc gets EINTR in one of its system calls (poll(2)). Second, nc treats EINTR as a serious error and exits. Looks like nc gets EINTR when dtrace does ptrace(PT_CONTINUE). So this is the culprit for your test case. Third, this is what happens on the dtrace side: Recorded dtrace behavior seems to be correct. 30495 100974 dtrace CALL write(0x1,0x8035c2000,0x1a) 30495 100974 dtrace GIO fd 1 wrote 26 bytes libc.so.7`__sys_accept+0xc 30495 100974 dtrace RET write 26/0x1a 30495 100974 dtrace CALL write(0x1,0x8035c2000,0x1) 30495 100974 dtrace GIO fd 1 wrote 1 byte 30495 100974 dtrace RET write 1 30495 100974 dtrace CALL thr_kill(0x1c129,SIGUSR1) 30495 100974 dtrace RET thr_kill 0 30495 100974 dtrace CALL ptrace(PT_IO,0x771e,0x7fffbb10,0) 30495 100974 dtrace RET ptrace 0 30495 100974 dtrace CALL ptrace(PT_IO,0x771e,0x7fffbb10,0) 30495 114985 dtrace RET _umtx_op -1 errno 4 Interrupted system call 30495 100974 dtrace RET ptrace 0 30495 114985 dtrace PSIG SIGUSR1 caught handler=0x80085d610 mask=0x5ffefedf code=0x10007 30495 100974 dtrace CALL _umtx_op(0x802007668,0x15,0x1,0,0) 30495 114985 dtrace CALL sigprocmask(SIG_SETMASK,0x7fbfd9fc,0) 30495 100974 dtrace RET _umtx_op 0 30495 114985 dtrace RET sigprocmask 0 30495 100974 dtrace CALL _umtx_op(0x800a71778,0xf,0,0,0) 30495 114985 dtrace CALL sigreturn(0x7fbfd630) 30495 114985 dtrace RET sigreturn JUSTRETURN 30495 114985 dtrace CALL ptrace(PT_CONTINUE,0x771e,0x1,0) 30495 114985 dtrace RET ptrace 0 30495 114985 dtrace CALL ptrace(PT_IO,0x771e,0x7fbfdee0,0) 30495 114985 dtrace RET ptrace -1 errno 16 Device busy 30495 114985 dtrace CALL _umtx_op(0x80200d668,0x15,0x1,0,0) 30495 114985 dtrace RET _umtx_op 0 30495 114985 dtrace CALL madvise(0x803c52000,0x11000,MADV_FREE) 30495 114985 dtrace RET madvise 0 30495 114985 dtrace CALL madvise(0x803c1d000,0x14000,MADV_FREE) 30495 114985 dtrace RET madvise 0 30495 114985 dtrace CALL madvise(0x803c07000,0x1000,MADV_FREE) 30495 114985 dtrace RET madvise 0 30495 114985 dtrace CALL madvise(0x8037fc000,0x2000,MADV_FREE) 30495 114985 dtrace RET madvise 0 30495 114985 dtrace CALL madvise(0x8037f2000,0x8000,MADV_FREE) 30495 114985 dtrace RET madvise 0 30495 114985 dtrace CALL madvise(0x8037f,0x1000,MADV_FREE) 30495 100974 dtrace RET _umtx_op 0 30495 114985 dtrace RET madvise 0 30495 100974 dtrace CALL ptrace(PT_DETACH,0x771e,0,0) 30495 114985 dtrace CALL madvise(0x8037e,0x9000,MADV_FREE) 30495 100974 dtrace RET ptrace -1 errno 3 No such process Looks like something fishy happens when that SIGUSR1 is generated. I wish a dtrace-for-userland expert and/or a ptrace(2) expert could help us here. -- Andriy Gapon ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: dtrace ustack kernel panic
Hi, just finished installing FreeBSD-9BETA1 and recompiling the kernel with dtrace. This is even worse, I have the same behavior as mentioned here: http://freebsd.1045724.n5.nabble.com/bin-158431-dtrace-crash-in-dt-proc-lookup-when-attaching-to-PID-assert-dpr-NULL-tt4535367.html#none i.e., dtrace regardless of whether with or without any probes just quits with the following error message Assertion failed: (dpr != NULL), file /usr/src/cddl/lib/libdtrace/../../../cddl/contrib/opensolaris/lib/libdtrace/common/dt_proc.c, line 751. that's inside dt_proc_lookup() I think I have to give up on ustack(), too bad cheers --m On Sat, Jul 30, 2011 at 2:05 PM, maestro something maestr...@gmail.comwrote: Hi, This is i386, right ? I think the cause is that assembler routine panic_trigger does not establish the standard i386 frame. Basically, you need either this, or dwarf annotations, for gdb to be able to walk over the frame. You need to add the standard prologue pushl %ebp movl%esp,%ebp and standard epilogue leave to the function. No idea whether it will continue to operate correctly after. my panic_trigger looks like this now: /* int panic_trigger(int *tp) */ ENTRY(panic_trigger) pushl %ebp movl%esp,%ebp xorl%eax, %eax movl$0xdefacedd, %edx lock xchgl %edx, (%edi) cmpl$0, %edx je 0f movl$0, %eax leave ret 0: movl$1, %eax leave ret END(panic_trigger) same result, (actually too same as the address in the stack trace is still the same, is that possible?) KDB: stack backtrace: #0 0xc09036a7 at kdb_backtrace+0x47 #1 0xc08d1a07 at panic+0x117 #2 0xc0c158c3 at trap_fatal+0x323 #3 0xc0c15bc0 at trap_pfault+0x2f0 #4 0xc0c1612a at trap+0x48a #5 0xc0bfc97c at calltrap+0x6 #6 0xc10e99db at dtrace_panic+0x1b #7 0xc10e9a0d at dtrace_assfail+0x2d #8 0xc10fa6a6 at dtrace_probe+0xfd6 #9 0xc1237ce4 at systrace_probe+0x84 #10 0xc090f63f at syscallenter+0x47f #11 0xc0c15c14 at syscall+0x34 #12 0xc0bfca11 at Xint0x80_syscall+0x21 I tried playing around with kgdb a bit. It finds source files until frame #10 (i.e., syscallenter + 0x47f) (kgdb) list *syscallenter+0x47f 0xc090f63f is in syscallenter (/usr/src/sys/kern/subr_trap.c:328). 323 * If the systrace module has registered it's probe 324 * callback and if there is a probe active for the 325 * syscall 'return', process the probe. 326 */ 327if (systrace_probe_func != NULL sa-callp-sy_return != 0) 328(*systrace_probe_func)(sa-callp-sy_return, sa-code, 329sa-callp, sa-args); 330#endif 331CTR4(KTR_SYSC, syscall: p=%p error=%d return %#lx %#lx, 332p, error, td-td_retval[0], td-td_retval[1]); (kgdb) list *systrace_probe+0x84 No source file for address 0xc1237ce4. Thus, it seems that the first frame where kgdb cannot find a source file is #9 (i.e., systrace_probe + 0x84) As far as I understand that's when the (imho) function pointer systrace_probe_func gets invoked. Therefore, it seems that kgdb has a hard time finding the source file for the function invoked through that pointer. Is this complete nonsense, or does that actually sound familiar to anyone? And still I'm wondering whether anybody ever successfully used the ustack action on Freebsd-8.2 i386 cheers --m ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: dtrace ustack kernel panic
Hi, This is even worse, I have the same behavior as mentioned here: http://freebsd.1045724.n5.nabble.com/bin-158431-dtrace-crash-in-dt-proc-lookup-when-attaching-to-PID-assert-dpr-NULL-tt4535367.html#none i.e., dtrace regardless of whether with or without any probes just quits with the following error message this is not accurate. It seems that this message only occurs if i use the ustack() action. i.e, dtrace -n 'syscall::accept:return /execname == nc / { printf(%s accept:return\n, execname);}' works as expected However, dtrace -n 'syscall::accept:return /execname == nc / { printf(%s accept:return\n, execname);ustack();}' does not: it results in the mentioned error message. Assertion failed: (dpr != NULL), file /usr/src/cddl/lib/libdtrace/../../../cddl/contrib/opensolaris/lib/libdtrace/common/dt_proc.c, line 751. cheers --m Assertion failed: (dpr != NULL), file /usr/src/cddl/lib/libdtrace/../../../cddl/contrib/opensolaris/lib/libdtrace/common/dt_proc.c, line 751. that's inside dt_proc_lookup() I think I have to give up on ustack(), too bad cheers --m On Sat, Jul 30, 2011 at 2:05 PM, maestro something maestr...@gmail.comwrote: Hi, This is i386, right ? I think the cause is that assembler routine panic_trigger does not establish the standard i386 frame. Basically, you need either this, or dwarf annotations, for gdb to be able to walk over the frame. You need to add the standard prologue pushl %ebp movl%esp,%ebp and standard epilogue leave to the function. No idea whether it will continue to operate correctly after. my panic_trigger looks like this now: /* int panic_trigger(int *tp) */ ENTRY(panic_trigger) pushl %ebp movl%esp,%ebp xorl%eax, %eax movl$0xdefacedd, %edx lock xchgl %edx, (%edi) cmpl$0, %edx je 0f movl$0, %eax leave ret 0: movl$1, %eax leave ret END(panic_trigger) same result, (actually too same as the address in the stack trace is still the same, is that possible?) KDB: stack backtrace: #0 0xc09036a7 at kdb_backtrace+0x47 #1 0xc08d1a07 at panic+0x117 #2 0xc0c158c3 at trap_fatal+0x323 #3 0xc0c15bc0 at trap_pfault+0x2f0 #4 0xc0c1612a at trap+0x48a #5 0xc0bfc97c at calltrap+0x6 #6 0xc10e99db at dtrace_panic+0x1b #7 0xc10e9a0d at dtrace_assfail+0x2d #8 0xc10fa6a6 at dtrace_probe+0xfd6 #9 0xc1237ce4 at systrace_probe+0x84 #10 0xc090f63f at syscallenter+0x47f #11 0xc0c15c14 at syscall+0x34 #12 0xc0bfca11 at Xint0x80_syscall+0x21 I tried playing around with kgdb a bit. It finds source files until frame #10 (i.e., syscallenter + 0x47f) (kgdb) list *syscallenter+0x47f 0xc090f63f is in syscallenter (/usr/src/sys/kern/subr_trap.c:328). 323 * If the systrace module has registered it's probe 324 * callback and if there is a probe active for the 325 * syscall 'return', process the probe. 326 */ 327if (systrace_probe_func != NULL sa-callp-sy_return != 0) 328(*systrace_probe_func)(sa-callp-sy_return, sa-code, 329sa-callp, sa-args); 330#endif 331CTR4(KTR_SYSC, syscall: p=%p error=%d return %#lx %#lx, 332p, error, td-td_retval[0], td-td_retval[1]); (kgdb) list *systrace_probe+0x84 No source file for address 0xc1237ce4. Thus, it seems that the first frame where kgdb cannot find a source file is #9 (i.e., systrace_probe + 0x84) As far as I understand that's when the (imho) function pointer systrace_probe_func gets invoked. Therefore, it seems that kgdb has a hard time finding the source file for the function invoked through that pointer. Is this complete nonsense, or does that actually sound familiar to anyone? And still I'm wondering whether anybody ever successfully used the ustack action on Freebsd-8.2 i386 cheers --m ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: dtrace ustack kernel panic
on 02/08/2011 20:46 maestro something said the following: Hi, just finished installing FreeBSD-9BETA1 and recompiling the kernel with dtrace. This is even worse, I have the same behavior as mentioned here: http://freebsd.1045724.n5.nabble.com/bin-158431-dtrace-crash-in-dt-proc-lookup-when-attaching-to-PID-assert-dpr-NULL-tt4535367.html#none Kind of a mentoring note: it would be much shorter and much more useful to paste PR 158431 or, even better, an http URL to the said PR in FreeBSD PR DB web interface. I know, I know, it's really Google to blame, right? :-) i.e., dtrace regardless of whether with or without any probes just quits with the following error message Assertion failed: (dpr != NULL), file /usr/src/cddl/lib/libdtrace/../../../cddl/contrib/opensolaris/lib/libdtrace/common/dt_proc.c, line 751. that's inside dt_proc_lookup() I think I have to give up on ustack(), too bad Or maybe you still have a chance to help us here, maybe it will even help you in the long term. It would be good if you pasted a little bit more of the output that you got. Please note that in that PR the (meaningful) output starts with: dtrace: no probes specified I.e. it starts with what dtrace considers to be a fatal condition. I tried to run dtruss (as you did) and I got this pre-amble before the assertion: [some dtrace script body] : probe description proc:::exit does not match any probes I guess that in my case I got it because my userland was not compiled with CTF support. Not sure about yours... Of course, it's still rather bad that dtrace crashes when it prematurely exits. But maybe it doesn't crash in the correct environment... I don't know. -- Andriy Gapon ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: dtrace ustack kernel panic
on 03/08/2011 00:03 Andriy Gapon said the following: I tried to run dtruss (as you did) and I got this pre-amble before the assertion: [some dtrace script body] : probe description proc:::exit does not match any probes I guess that in my case I got it because my userland was not compiled with CTF support. Not sure about yours... Of course, it's still rather bad that dtrace crashes when it prematurely exits. But maybe it doesn't crash in the correct environment... I don't know. OK, here's a patch that should fix the abort via assertion - with this you shouldn't get the crash that you've reported. I hope that you will be able to tackle other conditions that dtrace considers to be errors. diff --git a/lib/libproc/proc_create.c b/lib/libproc/proc_create.c index c372a47..9bd24a2 100644 --- a/lib/libproc/proc_create.c +++ b/lib/libproc/proc_create.c @@ -79,12 +79,11 @@ proc_attach(pid_t pid, int flags, struct proc_handle **pphdl) else phdl-status = PS_STOP; +out: if (error) proc_free(phdl); else *pphdl = phdl; -out: - proc_free(phdl); return (error); } -- Andriy Gapon ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: dtrace ustack kernel panic
Hi, http://freebsd.1045724.n5.nabble.com/bin-158431-dtrace-crash-in-dt-proc-lookup-when-attaching-to-PID-assert-dpr-NULL-tt4535367.html#none Kind of a mentoring note: it would be much shorter and much more useful to paste PR 158431 or, even better, an http URL to the said PR in FreeBSD PR DB web interface. I know, I know, it's really Google to blame, right? :-) True, google was showing this thread as a result to my search query. i.e., dtrace regardless of whether with or without any probes just quits with the following error message Assertion failed: (dpr != NULL), file /usr/src/cddl/lib/libdtrace/../../../cddl/contrib/opensolaris/lib/libdtrace/common/dt_proc.c, line 751. that's inside dt_proc_lookup() I think I have to give up on ustack(), too bad Or maybe you still have a chance to help us here, maybe it will even help you in the long term. It would be good if you pasted a little bit more of the output that you got. Please note that in that PR the (meaningful) output starts with: dtrace: no probes specified I.e. it starts with what dtrace considers to be a fatal condition. I tried to run dtruss (as you did) and I got this pre-amble before the assertion: [some dtrace script body] : probe description proc:::exit does not match any probes ok this is the command line and all the output i get from dtrace: fb90i386# dtrace -n 'syscall::accept:return / execname == nc / { @sc[execname,probefunc] = count(); printf(%s accept:return\n, execname); ustack();}' dtrace: description 'syscall::accept:return ' matched 1 probe CPU IDFUNCTION:NAME 0 46457accept:return nc accept:return Assertion failed: (dpr != NULL), file /usr/src/cddl/lib/libdtrace/../../../cddl/contrib/opensolaris/lib/libdtrace/common/dt_proc.c, line 751. Abort (core dumped) fb90i386# I get the same thing when using dtruss: fb90i386# dtruss -a -p 1139 Assertion failed: (dpr != NULL), file /usr/src/cddl/lib/libdtrace/../../../cddl/contrib/opensolaris/lib/libdtrace/common/dt_proc.c, line 751. Abort trap (core dumped) fb90i386# I guess that in my case I got it because my userland was not compiled with CTF support. Not sure about yours... hm userland and CTF not that I know. I only recompiled the kernel with CTF. Do I have to recompile userland (and would that be make world?) If so can i just pass the same WITH_CTF=1 option to make? Of course, it's still rather bad that dtrace crashes when it prematurely exits. But maybe it doesn't crash in the correct environment... I don't know. as soon as I understand what you mean by userland I can recompile the necessary bits. cheers and thanks a lot for your patience --m ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: dtrace ustack kernel panic
Hi, diff --git a/lib/libproc/proc_create.c b/lib/libproc/proc_create.c index c372a47..9bd24a2 100644 --- a/lib/libproc/proc_create.c +++ b/lib/libproc/proc_create.c @@ -79,12 +79,11 @@ proc_attach(pid_t pid, int flags, struct proc_handle **pphdl) else phdl-status = PS_STOP; +out: if (error) proc_free(phdl); else *pphdl = phdl; -out: - proc_free(phdl); return (error); } What do I have to recompile for the patch to be picked up? make libproc in /usr/src/lib works but make install libproc fails the following way: fb90i386# make libproc Warning: Object directory not changed from original /usr/src/lib/libproc cc -O2 -pipe -I/usr/src/lib/libproc -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -c proc_bkpt.c cc -O2 -pipe -I/usr/src/lib/libproc -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -c proc_create.c cc -O2 -pipe -I/usr/src/lib/libproc -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -c proc_regs.c cc -O2 -pipe -I/usr/src/lib/libproc -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -c proc_sym.c cc -O2 -pipe -I/usr/src/lib/libproc -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -c proc_rtld.c cc -O2 -pipe -I/usr/src/lib/libproc -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -c proc_util.c building static proc library ranlib libproc.a cc -pg -O2 -pipe -I/usr/src/lib/libproc -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -c proc_bkpt.c -o proc_bkpt.po cc -pg -O2 -pipe -I/usr/src/lib/libproc -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -c proc_create.c -o proc_create.po cc -pg -O2 -pipe -I/usr/src/lib/libproc -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -c proc_regs.c -o proc_regs.po cc -pg -O2 -pipe -I/usr/src/lib/libproc -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -c proc_sym.c -o proc_sym.po cc -pg -O2 -pipe -I/usr/src/lib/libproc -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall
Re: dtrace ustack kernel panic
On Tue, Aug 2, 2011 at 3:46 PM, maestro something maestr...@gmail.com wrote: Hi, diff --git a/lib/libproc/proc_create.c b/lib/libproc/proc_create.c index c372a47..9bd24a2 100644 --- a/lib/libproc/proc_create.c +++ b/lib/libproc/proc_create.c @@ -79,12 +79,11 @@ proc_attach(pid_t pid, int flags, struct proc_handle **pphdl) else phdl-status = PS_STOP; +out: if (error) proc_free(phdl); else *pphdl = phdl; -out: - proc_free(phdl); return (error); } What do I have to recompile for the patch to be picked up? make libproc in /usr/src/lib works but make install libproc fails the following way: fb90i386# make install libproc === csu/i386-elf (install) install -o root -g wheel -m 444 crti.o crtn.o gcrt1.o crt1.o Scrt1.o /usr/lib === libc (install) install -C -o root -g wheel -m 444 libc.a /usr/lib install: libc.a: No such file or directory *** Error code 71 Stop in /usr/src/lib/libc. *** Error code 1 Stop in /usr/src/lib. Did you remember to 'make obj'? If you didn't, the library would be left in the wrong place where 'make install' would not find it. make obj make depend (possibly a no-op) make make install -- R. Kevin Oberman, Network Engineer - Retired E-mail: kob6...@gmail.com ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: dtrace ustack kernel panic
Hi, Hi, diff --git a/lib/libproc/proc_create.c b/lib/libproc/proc_create.c index c372a47..9bd24a2 100644 --- a/lib/libproc/proc_create.c +++ b/lib/libproc/proc_create.c @@ -79,12 +79,11 @@ proc_attach(pid_t pid, int flags, struct proc_handle **pphdl) else phdl-status = PS_STOP; +out: if (error) proc_free(phdl); else *pphdl = phdl; -out: - proc_free(phdl); return (error); } What do I have to recompile for the patch to be picked up? make libproc in /usr/src/lib works but make install libproc fails the following way: fb90i386# make install libproc === csu/i386-elf (install) install -o root -g wheel -m 444 crti.o crtn.o gcrt1.o crt1.o Scrt1.o /usr/lib === libc (install) install -C -o root -g wheel -m 444 libc.a /usr/lib install: libc.a: No such file or directory *** Error code 71 Stop in /usr/src/lib/libc. *** Error code 1 Stop in /usr/src/lib. Did you remember to 'make obj'? If you didn't, the library would be left in the wrong place where 'make install' would not find it. make obj make depend (possibly a no-op) make make install Thank you very much that helped. now it seems that the ustack() action on i386 and FreeBSD-9.0BETA1 almost works. Almost because dtrace still exits (this time without an error message) The process for which the ustack should be genereated is terminated Here are the specifics: fb90i386# dtrace -n 'syscall::accept:return / execname == nc / { ustack();}' dtrace: description 'syscall::accept:return ' matched 1 probe CPU IDFUNCTION:NAME 0 46457accept:return libc.so.7`__sys_accept+0x7 0x3 0xed96e824 This alsmost looks like a stack trace I'd say However, dtrace quits once I connect to nc nc quits with the following error message fb90i386# nc -vl nc: Polling Error: Interrupted system call cheers -m ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: dtrace ustack kernel panic
Running the same example on amd64 has the same behavior as on i386 (granted the backtrace is slightly different). That is, the kernel panics immediately when you use the ustack() action. Furthermore, the crashdump does not provide the nececcary information to get at the root of the cause (i.e., no souce found for the systrace_probe and dtrace_probe frames). still I'm wondering ... Did anybody successfully use the ustack() action on FreeBSD-8.2? cheers --m backtrace on amd64: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address= 0x0 fault code= supervisor read data, page not present instruction pointer= 0x20:0x80ff1fa6 stack pointer= 0x28:0xff80001d7500 frame pointer= 0x28:0xff80001d7850 code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags= resume, IOPL = 0 current process= 1068 (nc) trap number= 12 panic: page fault cpuid = 0 KDB: stack backtrace: #0 0x80618d2e at kdb_backtrace+0x5e #1 0x805e4d37 at panic+0x187 #2 0x808dcac0 at trap_fatal+0x290 #3 0x808dcf0d at trap_pfault+0x2fd #4 0x808dd343 at trap+0x343 #5 0x808c5298 at calltrap+0x8 #6 0x80ff638a at dtrace_probe+0x47a #7 0x81192285 at systrace_probe+0x45 #8 0x80624f50 at syscallenter+0x3c0 #9 0x808dcf7b at syscall+0x4b #10 0x808c5572 at Xfast_syscall+0xe2 Uptime: 3m0s On Sun, Jul 31, 2011 at 6:46 AM, Alex Samorukov m...@os2.kiev.ua wrote: Hi, I just tried your test on -STABLE + 1 fix from current [1] and got no trap. dtrace: buffer size lowered to 2m CPU IDFUNCTION:NAME 4 42224accept:return nc accept:return Assertion failed: (dpr != NULL), file /usr/src/cddl/lib/libdtrace/..** /../../cddl/contrib/**opensolaris/lib/libdtrace/**common/dt_proc.c, line 751. Abort (core dumped) Also 1st dtrace was killed as well. I tried 2 times with the same result. Box is amd64. This is bt from gdb: (gdb) bt #0 0x000800fa2f9c in thr_kill () from /lib/libc.so.7 #1 0x00080103f6cb in abort () from /lib/libc.so.7 #2 0x000800786758 in dt_proc_lookup () from /lib/libdtrace.so.2 #3 0x0008007867db in dt_proc_lock () from /lib/libdtrace.so.2 #4 0x0008007a5bcb in dt_print_ustack () from /lib/libdtrace.so.2 #5 0x0008007a8504 in dt_print_agg () from /lib/libdtrace.so.2 #6 0x0008007a88ca in dtrace_consume () from /lib/libdtrace.so.2 #7 0x00080077e1a2 in dtrace_work () from /lib/libdtrace.so.2 #8 0x004044ba in ?? () #9 0x0040222e in ?? () [skip] [1] http://www.freebsd.org/cgi/**query-pr.cgi?pr=kern/159064http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/159064 On 07/26/2011 03:20 AM, maestro something wrote: Hi, when using the ustack action on the latest FreeBSD8.2 i386 the kernel panics. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: dtrace ustack kernel panic
Hi, I just tried your test on -STABLE + 1 fix from current [1] and got no trap. dtrace: buffer size lowered to 2m CPU IDFUNCTION:NAME 4 42224accept:return nc accept:return Assertion failed: (dpr != NULL), file /usr/src/cddl/lib/libdtrace/../../../cddl/contrib/opensolaris/lib/libdtrace/common/dt_proc.c, line 751. Abort (core dumped) Also 1st dtrace was killed as well. I tried 2 times with the same result. Box is amd64. This is bt from gdb: (gdb) bt #0 0x000800fa2f9c in thr_kill () from /lib/libc.so.7 #1 0x00080103f6cb in abort () from /lib/libc.so.7 #2 0x000800786758 in dt_proc_lookup () from /lib/libdtrace.so.2 #3 0x0008007867db in dt_proc_lock () from /lib/libdtrace.so.2 #4 0x0008007a5bcb in dt_print_ustack () from /lib/libdtrace.so.2 #5 0x0008007a8504 in dt_print_agg () from /lib/libdtrace.so.2 #6 0x0008007a88ca in dtrace_consume () from /lib/libdtrace.so.2 #7 0x00080077e1a2 in dtrace_work () from /lib/libdtrace.so.2 #8 0x004044ba in ?? () #9 0x0040222e in ?? () [skip] [1] http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/159064 On 07/26/2011 03:20 AM, maestro something wrote: Hi, when using the ustack action on the latest FreeBSD8.2 i386 the kernel panics. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: dtrace ustack kernel panic
on 30/07/2011 00:27 maestro something said the following: Hi, trying to do so I don't really find my way around. This is what I get when I run kgdb On startup the assert frame is #7 and the probe frame is #8. [snip] kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address= 0x108 fault code= supervisor write, page not present instruction pointer= 0x20:0xc11012d7 stack pointer= 0x28:0xcd3ed9f4 frame pointer= 0x28:0xcd3eda0c code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags= resume, IOPL = 0 current process= 1132 (nc) trap number= 12 panic: page fault cpuid = 0 KDB: stack backtrace: #0 0xc09036a7 at kdb_backtrace+0x47 #1 0xc08d1a07 at panic+0x117 #2 0xc0c158c3 at trap_fatal+0x323 #3 0xc0c15bc0 at trap_pfault+0x2f0 #4 0xc0c1612a at trap+0x48a #5 0xc0bfc97c at calltrap+0x6 #6 0xc10e992b at dtrace_panic+0x1b #7 0xc10e995d at dtrace_assfail+0x2d #8 0xc10fb28c at dtrace_probe+0x135c #9 0xc1237f72 at systrace_probe+0x62 #10 0xc090f63f at syscallenter+0x47f #11 0xc0c15c14 at syscall+0x34 #12 0xc0bfca11 at Xint0x80_syscall+0x21 Uptime: 3m0s Physical memory: 239 MB Dumping 79 MB: 64 48 32 16 [snip] #0 doadump () at pcpu.h:231 231__asm(movl %%fs:0,%0 : =r (td)); once I'm in the kgdb console i type where and all of a sudden the stack trace has only 7 frames... that does not seem correct. Furthermore, the Previous frame inner to this frame (corrupt stack?) error message is not too encuraging either. (kgdb) where #0 doadump () at pcpu.h:231 #1 0xc08d17a3 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:419 #2 0xc08d1a40 in panic (fmt=Variable fmt is not available. ) at /usr/src/sys/kern/kern_shutdown.c:592 #3 0xc0c158c3 in trap_fatal (frame=0xcd3ed9b4, eva=264) at /usr/src/sys/i386/i386/trap.c:946 #4 0xc0c15bc0 in trap_pfault (frame=0xcd3ed9b4, usermode=0, eva=264) at /usr/src/sys/i386/i386/trap.c:859 #5 0xc0c1612a in trap (frame=0xcd3ed9b4) at /usr/src/sys/i386/i386/trap.c:532 #6 0xc0bfc97c in calltrap () at /usr/src/sys/i386/i386/exception.s:166 #7 0xc11012d7 in dtrace_panic_trigger () from /boot/kernel/dtrace.ko Previous frame inner to this frame (corrupt stack?) (kgdb) what am I doing wrong and what do I have to do to get the specific assert that fails? I am not quite sure. Apparently there is some issue with either what information we store in a crash dump and how, or with how kgdb interprets that information, or with a combination of both... Perhaps this is a result of -O2 optimization and -O1 would improve the situation - not sure again... Meanwhile, there is something simple that you can do without much hassle: (kgdb) list *dtrace_probe+0x135c where the address is taken from the backtrace printed by panic(9). On Tue, Jul 26, 2011 at 4:05 AM, Andriy Gapon a...@freebsd.org mailto:a...@freebsd.org wrote: on 26/07/2011 04:20 maestro something said the following: Hi, when using the ustack action on the latest FreeBSD8.2 i386 the kernel panics. Here is the information I could gather: let me know if I can provide more information. (i.e., i have the full crash information 80+mbs handy) Use kgdb on the crash dump, go to the dtrace_probe frame and see which exactly assert fails and why. Here is the backtrace: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x108 fault code = supervisor write, page not present instruction pointer = 0x20:0xc11012d7 stack pointer = 0x28:0xcd3ed9f4 frame pointer = 0x28:0xcd3eda0c code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags= resume, IOPL = 0 current process = 1132 (nc) trap number = 12 panic: page fault cpuid = 0 KDB: stack backtrace: #0 0xc09036a7 at kdb_backtrace+0x47 #1 0xc08d1a07 at panic+0x117 #2 0xc0c158c3 at trap_fatal+0x323 #3 0xc0c15bc0 at trap_pfault+0x2f0 #4 0xc0c1612a at trap+0x48a #5 0xc0bfc97c at calltrap+0x6 #6 0xc10e992b at dtrace_panic+0x1b #7 0xc10e995d at dtrace_assfail+0x2d #8 0xc10fb28c at dtrace_probe+0x135c #9 0xc1237f72 at systrace_probe+0x62 #10 0xc090f63f at syscallenter+0x47f #11 0xc0c15c14 at syscall+0x34 #12 0xc0bfca11 at Xint0x80_syscall+0x21 Uptime: 3m0s Physical memory: 239 MB Dumping 79 MB: 64 48 32 16 Steps To reproduce: the dtrace program (i.e., test.d) was: #!/usr/sbin/dtrace -s syscall::accept:return / execname == nc /
Re: dtrace ustack kernel panic
on 30/07/2011 00:27 maestro something said the following: on 30/07/2011 10:50 Andriy Gapon said the following: #7 0xc11012d7 in dtrace_panic_trigger () from /boot/kernel/dtrace.ko Previous frame inner to this frame (corrupt stack?) (kgdb) what am I doing wrong and what do I have to do to get the specific assert that fails? I am not quite sure. Apparently there is some issue with either what information we store in a crash dump and how, or with how kgdb interprets that information, or with a combination of both... Or maybe it's because dtrace_panic_trigger is a function implemented in assembly and we do not provide enough debug information to (k)gdb in that case. -- Andriy Gapon ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: dtrace ustack kernel panic
On Sat, Jul 30, 2011 at 12:50 AM, Andriy Gapon a...@freebsd.org wrote: on 30/07/2011 00:27 maestro something said the following: Hi, trying to do so I don't really find my way around. This is what I get when I run kgdb On startup the assert frame is #7 and the probe frame is #8. [snip] kernel trap 12 with interrupts disabled I am not quite sure. Apparently there is some issue with either what information we store in a crash dump and how, or with how kgdb interprets that information, or with a combination of both... Perhaps this is a result of -O2 optimization and -O1 would improve the situation I guess I have to recompile the kernel without -O then ... does that work for freebsd (I remember linux has issues i.e., does not compile without any -O) How would I do that the in the most straight forward way? Meanwhile, there is something simple that you can do without much hassle: (kgdb) list *dtrace_probe+0x135c True there was not a lot hassle involved. However the result is not really satisfactory either :-) (kgdb) list *dtrace_probe+0x135c No source file for address 0xc10fb28c. The address is in accordance with the stack-trace (i.e., frame #8) where the address is taken from the backtrace printed by panic(9). cheers --m On Tue, Jul 26, 2011 at 4:05 AM, Andriy Gapon a...@freebsd.org mailto:a...@freebsd.org wrote: on 26/07/2011 04:20 maestro something said the following: Hi, when using the ustack action on the latest FreeBSD8.2 i386 the kernel panics. Here is the information I could gather: let me know if I can provide more information. (i.e., i have the full crash information 80+mbs handy) Use kgdb on the crash dump, go to the dtrace_probe frame and see which exactly assert fails and why. Here is the backtrace: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x108 fault code = supervisor write, page not present instruction pointer = 0x20:0xc11012d7 stack pointer = 0x28:0xcd3ed9f4 frame pointer = 0x28:0xcd3eda0c code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags= resume, IOPL = 0 current process = 1132 (nc) trap number = 12 panic: page fault cpuid = 0 KDB: stack backtrace: #0 0xc09036a7 at kdb_backtrace+0x47 #1 0xc08d1a07 at panic+0x117 #2 0xc0c158c3 at trap_fatal+0x323 #3 0xc0c15bc0 at trap_pfault+0x2f0 #4 0xc0c1612a at trap+0x48a #5 0xc0bfc97c at calltrap+0x6 #6 0xc10e992b at dtrace_panic+0x1b #7 0xc10e995d at dtrace_assfail+0x2d #8 0xc10fb28c at dtrace_probe+0x135c #9 0xc1237f72 at systrace_probe+0x62 #10 0xc090f63f at syscallenter+0x47f #11 0xc0c15c14 at syscall+0x34 #12 0xc0bfca11 at Xint0x80_syscall+0x21 Uptime: 3m0s Physical memory: 239 MB Dumping 79 MB: 64 48 32 16 Steps To reproduce: the dtrace program (i.e., test.d) was: #!/usr/sbin/dtrace -s syscall::accept:return / execname == nc / { printf(%s accept:return\n, execname); ustack(); } % dtrace -s test.d then running % nc -vl 8080 on one shell and % nc localhost 8080 on another would make the kernel panic -- Andriy Gapon ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: dtrace ustack kernel panic
on 30/07/2011 21:19 maestro something said the following: On Sat, Jul 30, 2011 at 12:50 AM, Andriy Gapon a...@freebsd.org mailto:a...@freebsd.org wrote: on 30/07/2011 00:27 maestro something said the following: Hi, trying to do so I don't really find my way around. This is what I get when I run kgdb On startup the assert frame is #7 and the probe frame is #8. [snip] kernel trap 12 with interrupts disabled I am not quite sure. Apparently there is some issue with either what information we store in a crash dump and how, or with how kgdb interprets that information, or with a combination of both... Perhaps this is a result of -O2 optimization and -O1 would improve the situation I guess I have to recompile the kernel without -O then ... does that work for freebsd (I remember linux has issues i.e., does not compile without any -O) Not sure about -O0; -O1/-O should work fine. How would I do that the in the most straight forward way? See man make.conf: CFLAGS, etc. Meanwhile, there is something simple that you can do without much hassle: (kgdb) list *dtrace_probe+0x135c True there was not a lot hassle involved. However the result is not really satisfactory either :-) (kgdb) list *dtrace_probe+0x135c No source file for address 0xc10fb28c. The address is in accordance with the stack-trace (i.e., frame #8) where the address is taken from the backtrace printed by panic(9). Have you started kgdb with the correct kernel and core file? If yes, then I am out of ideas. On Tue, Jul 26, 2011 at 4:05 AM, Andriy Gapon a...@freebsd.org mailto:a...@freebsd.org mailto:a...@freebsd.org mailto:a...@freebsd.org wrote: on 26/07/2011 04:20 maestro something said the following: Hi, when using the ustack action on the latest FreeBSD8.2 i386 the kernel panics. Here is the information I could gather: let me know if I can provide more information. (i.e., i have the full crash information 80+mbs handy) Use kgdb on the crash dump, go to the dtrace_probe frame and see which exactly assert fails and why. Here is the backtrace: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x108 fault code = supervisor write, page not present instruction pointer = 0x20:0xc11012d7 stack pointer = 0x28:0xcd3ed9f4 frame pointer = 0x28:0xcd3eda0c code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags= resume, IOPL = 0 current process = 1132 (nc) trap number = 12 panic: page fault cpuid = 0 KDB: stack backtrace: #0 0xc09036a7 at kdb_backtrace+0x47 #1 0xc08d1a07 at panic+0x117 #2 0xc0c158c3 at trap_fatal+0x323 #3 0xc0c15bc0 at trap_pfault+0x2f0 #4 0xc0c1612a at trap+0x48a #5 0xc0bfc97c at calltrap+0x6 #6 0xc10e992b at dtrace_panic+0x1b #7 0xc10e995d at dtrace_assfail+0x2d #8 0xc10fb28c at dtrace_probe+0x135c #9 0xc1237f72 at systrace_probe+0x62 #10 0xc090f63f at syscallenter+0x47f #11 0xc0c15c14 at syscall+0x34 #12 0xc0bfca11 at Xint0x80_syscall+0x21 Uptime: 3m0s Physical memory: 239 MB Dumping 79 MB: 64 48 32 16 Steps To reproduce: the dtrace program (i.e., test.d) was: #!/usr/sbin/dtrace -s syscall::accept:return / execname == nc / { printf(%s accept:return\n, execname); ustack(); } % dtrace -s test.d then running % nc -vl 8080 on one shell and % nc localhost 8080 on another would make the kernel panic -- Andriy Gapon -- Andriy Gapon ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: dtrace ustack kernel panic
On Sat, Jul 30, 2011 at 11:27 AM, Andriy Gapon a...@freebsd.org wrote: on 30/07/2011 21:19 maestro something said the following: On Sat, Jul 30, 2011 at 12:50 AM, Andriy Gapon a...@freebsd.org mailto:a...@freebsd.org wrote: on 30/07/2011 00:27 maestro something said the following: Hi, trying to do so I don't really find my way around. This is what I get when I run kgdb On startup the assert frame is #7 and the probe frame is #8. [snip] kernel trap 12 with interrupts disabled I am not quite sure. Apparently there is some issue with either what information we store in a crash dump and how, or with how kgdb interprets that information, or with a combination of both... Perhaps this is a result of -O2 optimization and -O1 would improve the situation I guess I have to recompile the kernel without -O then ... does that work for freebsd (I remember linux has issues i.e., does not compile without any -O) Not sure about -O0; -O1/-O should work fine. How would I do that the in the most straight forward way? See man make.conf: CFLAGS, etc. Looking into that as I type. Meanwhile, there is something simple that you can do without much hassle: (kgdb) list *dtrace_probe+0x135c True there was not a lot hassle involved. However the result is not really satisfactory either :-) (kgdb) list *dtrace_probe+0x135c No source file for address 0xc10fb28c. The address is in accordance with the stack-trace (i.e., frame #8) where the address is taken from the backtrace printed by panic(9). Have you started kgdb with the correct kernel and core file? If yes, then I am out of ideas. I hope so, I only recompiled the kernel once according to the DTRACE wiki instructions and I certainly only have one /var/crash/vmcore.* file. I'll try recompiling the kernel with -O1 and try again. In the meantime, I'm wondering whether I'm really the only/first one that ran into this problem or if there are people that actually successfully used the ustack() target on freebsd-8.2? cheers --m On Tue, Jul 26, 2011 at 4:05 AM, Andriy Gapon a...@freebsd.org mailto:a...@freebsd.org mailto:a...@freebsd.org mailto:a...@freebsd.org wrote: on 26/07/2011 04:20 maestro something said the following: Hi, when using the ustack action on the latest FreeBSD8.2 i386 the kernel panics. Here is the information I could gather: let me know if I can provide more information. (i.e., i have the full crash information 80+mbs handy) Use kgdb on the crash dump, go to the dtrace_probe frame and see which exactly assert fails and why. Here is the backtrace: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x108 fault code = supervisor write, page not present instruction pointer = 0x20:0xc11012d7 stack pointer = 0x28:0xcd3ed9f4 frame pointer = 0x28:0xcd3eda0c code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags= resume, IOPL = 0 current process = 1132 (nc) trap number = 12 panic: page fault cpuid = 0 KDB: stack backtrace: #0 0xc09036a7 at kdb_backtrace+0x47 #1 0xc08d1a07 at panic+0x117 #2 0xc0c158c3 at trap_fatal+0x323 #3 0xc0c15bc0 at trap_pfault+0x2f0 #4 0xc0c1612a at trap+0x48a #5 0xc0bfc97c at calltrap+0x6 #6 0xc10e992b at dtrace_panic+0x1b #7 0xc10e995d at dtrace_assfail+0x2d #8 0xc10fb28c at dtrace_probe+0x135c #9 0xc1237f72 at systrace_probe+0x62 #10 0xc090f63f at syscallenter+0x47f #11 0xc0c15c14 at syscall+0x34 #12 0xc0bfca11 at Xint0x80_syscall+0x21 Uptime: 3m0s Physical memory: 239 MB Dumping 79 MB: 64 48 32 16 Steps To reproduce: the dtrace program (i.e., test.d) was: #!/usr/sbin/dtrace -s syscall::accept:return / execname == nc / { printf(%s accept:return\n, execname); ustack(); } % dtrace -s test.d then running % nc -vl 8080 on one shell and % nc localhost 8080 on another would make the kernel panic -- Andriy Gapon
Re: dtrace ustack kernel panic
Hi, Have you started kgdb with the correct kernel and core file? If yes, then I am out of ideas. I hope so, I only recompiled the kernel once according to the DTRACE wiki instructions and I certainly only have one /var/crash/vmcore.* file. I'll try recompiling the kernel with -O1 and try again. In the meantime, I'm wondering whether I'm really the only/first one that ran into this problem or if there are people that actually successfully used the ustack() target on freebsd-8.2? I could not get the information even after recompiling the kernel here is the relevant (I think information). fb82i386# cat /etc/make.conf CFLAGS= -O (accodring to man make.conf only -O and -O2 is supported for CFLAGS anyways) kernel.debug is the newly compiled kernel (according to the timestamp) fb82i386# kgdb kernel.debug /var/crash/vmcore.0 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type show copying to see the conditions. There is absolutely no warranty for GDB. Type show warranty for details. This GDB was configured as i386-marcel-freebsd... Unread portion of the kernel message buffer: kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address= 0x108 fault code= supervisor write, page not present instruction pointer= 0x20:0xc1100847 stack pointer= 0x28:0xcd39a9e4 frame pointer= 0x28:0xcd39a9fc code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags= resume, IOPL = 0 current process= 1060 (nc) trap number= 12 panic: page fault cpuid = 0 KDB: stack backtrace: #0 0xc09036a7 at kdb_backtrace+0x47 #1 0xc08d1a07 at panic+0x117 #2 0xc0c158c3 at trap_fatal+0x323 #3 0xc0c15bc0 at trap_pfault+0x2f0 #4 0xc0c1612a at trap+0x48a #5 0xc0bfc97c at calltrap+0x6 #6 0xc10e99db at dtrace_panic+0x1b #7 0xc10e9a0d at dtrace_assfail+0x2d #8 0xc10fa6a6 at dtrace_probe+0xfd6 #9 0xc1237ce4 at systrace_probe+0x84 #10 0xc090f63f at syscallenter+0x47f #11 0xc0c15c14 at syscall+0x34 #12 0xc0bfca11 at Xint0x80_syscall+0x21 Uptime: 2m39s Physical memory: 239 MB Dumping 78 MB: 63 47 31 15 (kgdb) where #0 doadump () at pcpu.h:231 #1 0xc08d17a3 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:419 #2 0xc08d1a40 in panic (fmt=Variable fmt is not available. ) at /usr/src/sys/kern/kern_shutdown.c:592 #3 0xc0c158c3 in trap_fatal (frame=0xcd39a9a4, eva=264) at /usr/src/sys/i386/i386/trap.c:946 #4 0xc0c15bc0 in trap_pfault (frame=0xcd39a9a4, usermode=0, eva=264) at /usr/src/sys/i386/i386/trap.c:859 #5 0xc0c1612a in trap (frame=0xcd39a9a4) at /usr/src/sys/i386/i386/trap.c:532 #6 0xc0bfc97c in calltrap () at /usr/src/sys/i386/i386/exception.s:166 #7 0xc1100847 in dtrace_panic_trigger () from /boot/kernel/dtrace.ko Previous frame inner to this frame (corrupt stack?) (kgdb) list *dtrace_probe+0xfd6 No source file for address 0xc10fa6a6. So I'm stuck at the same point. any other ideas? cheers --m cheers --m On Tue, Jul 26, 2011 at 4:05 AM, Andriy Gapon a...@freebsd.org mailto:a...@freebsd.org mailto:a...@freebsd.org mailto:a...@freebsd.org wrote: on 26/07/2011 04:20 maestro something said the following: Hi, when using the ustack action on the latest FreeBSD8.2 i386 the kernel panics. Here is the information I could gather: let me know if I can provide more information. (i.e., i have the full crash information 80+mbs handy) Use kgdb on the crash dump, go to the dtrace_probe frame and see which exactly assert fails and why. Here is the backtrace: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x108 fault code = supervisor write, page not present instruction pointer = 0x20:0xc11012d7 stack pointer = 0x28:0xcd3ed9f4 frame pointer = 0x28:0xcd3eda0c code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags= resume, IOPL = 0 current process = 1132 (nc) trap number = 12 panic: page fault cpuid = 0 KDB: stack backtrace: #0 0xc09036a7 at kdb_backtrace+0x47 #1 0xc08d1a07 at panic+0x117 #2 0xc0c158c3 at trap_fatal+0x323 #3 0xc0c15bc0 at trap_pfault+0x2f0 #4 0xc0c1612a at trap+0x48a #5 0xc0bfc97c at
Re: dtrace ustack kernel panic
On Sat, Jul 30, 2011 at 12:05:33PM -0700, maestro something wrote: Hi, Have you started kgdb with the correct kernel and core file? If yes, then I am out of ideas. I hope so, I only recompiled the kernel once according to the DTRACE wiki instructions and I certainly only have one /var/crash/vmcore.* file. I'll try recompiling the kernel with -O1 and try again. In the meantime, I'm wondering whether I'm really the only/first one that ran into this problem or if there are people that actually successfully used the ustack() target on freebsd-8.2? I could not get the information even after recompiling the kernel here is the relevant (I think information). fb82i386# cat /etc/make.conf CFLAGS= -O (accodring to man make.conf only -O and -O2 is supported for CFLAGS anyways) kernel.debug is the newly compiled kernel (according to the timestamp) fb82i386# kgdb kernel.debug /var/crash/vmcore.0 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type show copying to see the conditions. There is absolutely no warranty for GDB. Type show warranty for details. This GDB was configured as i386-marcel-freebsd... Unread portion of the kernel message buffer: kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address= 0x108 fault code= supervisor write, page not present instruction pointer= 0x20:0xc1100847 stack pointer= 0x28:0xcd39a9e4 frame pointer= 0x28:0xcd39a9fc code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags= resume, IOPL = 0 current process= 1060 (nc) trap number= 12 panic: page fault cpuid = 0 KDB: stack backtrace: #0 0xc09036a7 at kdb_backtrace+0x47 #1 0xc08d1a07 at panic+0x117 #2 0xc0c158c3 at trap_fatal+0x323 #3 0xc0c15bc0 at trap_pfault+0x2f0 #4 0xc0c1612a at trap+0x48a #5 0xc0bfc97c at calltrap+0x6 #6 0xc10e99db at dtrace_panic+0x1b #7 0xc10e9a0d at dtrace_assfail+0x2d #8 0xc10fa6a6 at dtrace_probe+0xfd6 #9 0xc1237ce4 at systrace_probe+0x84 #10 0xc090f63f at syscallenter+0x47f #11 0xc0c15c14 at syscall+0x34 #12 0xc0bfca11 at Xint0x80_syscall+0x21 Uptime: 2m39s Physical memory: 239 MB Dumping 78 MB: 63 47 31 15 (kgdb) where #0 doadump () at pcpu.h:231 #1 0xc08d17a3 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:419 #2 0xc08d1a40 in panic (fmt=Variable fmt is not available. ) at /usr/src/sys/kern/kern_shutdown.c:592 #3 0xc0c158c3 in trap_fatal (frame=0xcd39a9a4, eva=264) at /usr/src/sys/i386/i386/trap.c:946 #4 0xc0c15bc0 in trap_pfault (frame=0xcd39a9a4, usermode=0, eva=264) at /usr/src/sys/i386/i386/trap.c:859 #5 0xc0c1612a in trap (frame=0xcd39a9a4) at /usr/src/sys/i386/i386/trap.c:532 #6 0xc0bfc97c in calltrap () at /usr/src/sys/i386/i386/exception.s:166 #7 0xc1100847 in dtrace_panic_trigger () from /boot/kernel/dtrace.ko Previous frame inner to this frame (corrupt stack?) (kgdb) list *dtrace_probe+0xfd6 No source file for address 0xc10fa6a6. So I'm stuck at the same point. any other ideas? This is i386, right ? I think the cause is that assembler routine panic_trigger does not establish the standard i386 frame. Basically, you need either this, or dwarf annotations, for gdb to be able to walk over the frame. You need to add the standard prologue pushl %ebp movl%esp,%ebp and standard epilogue leave to the function. No idea whether it will continue to operate correctly after. pgpTmrHqEp6po.pgp Description: PGP signature
Re: dtrace ustack kernel panic
on 30/07/2011 22:05 maestro something said the following: fb82i386# cat /etc/make.conf CFLAGS= -O (accodring to man make.conf only -O and -O2 is supported for CFLAGS anyways) kernel.debug is the newly compiled kernel (according to the timestamp) fb82i386# kgdb kernel.debug /var/crash/vmcore.0 Is vmcore.0 also new (i.e. produced with the new kernel)? Does 'list ...' command still produces no useful output? P.S. Also, hmm, I think that you shouldn't run kgdb on kernel.debug, just to avoid any confusion... I think that you should run something like kgdb /boot/${your installed kernel}/kernel /var/crash/vmcore.0; the debug symbols are automatically picked up from *.symbols files in the kernel directory. -- Andriy Gapon ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: dtrace ustack kernel panic
Hi, On Sat, Jul 30, 2011 at 12:52 PM, Andriy Gapon a...@freebsd.org wrote: on 30/07/2011 22:05 maestro something said the following: fb82i386# cat /etc/make.conf CFLAGS= -O (accodring to man make.conf only -O and -O2 is supported for CFLAGS anyways) kernel.debug is the newly compiled kernel (according to the timestamp) fb82i386# kgdb kernel.debug /var/crash/vmcore.0 Is vmcore.0 also new (i.e. produced with the new kernel)? Does 'list ...' command still produces no useful output? As mentioned before, the list command still gives the same error message: P.S. Also, hmm, I think that you shouldn't run kgdb on kernel.debug, just to avoid any confusion... I think that you should run something like kgdb /boot/${your installed kernel}/kernel /var/crash/vmcore.0; the debug symbols are automatically picked up from *.symbols files in the kernel directory. Same result when using /boot/kernel/kernel (kgdb) list *dtrace_probe+0xfd6 No source file for address 0xc10fa6a6. cheers --m ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: dtrace ustack kernel panic
on 30/07/2011 23:03 maestro something said the following: (kgdb) list *dtrace_probe+0xfd6 No source file for address 0xc10fa6a6. Are the sources on the same machine? This is probably the last idea from me. -- Andriy Gapon ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: dtrace ustack kernel panic
Hi, on 30/07/2011 23:03 maestro something said the following: (kgdb) list *dtrace_probe+0xfd6 No source file for address 0xc10fa6a6. Are the sources on the same machine? This is probably the last idea from me. Yes all is done on the same (virtual 32bit) machine. Sources are there too. If i do make installkernel KERNCONF=MYKERNEL the kernel should be installed right? (I assume so b/c /boot/kernel/kernel has the correct new timestamp) cheers -m ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: dtrace ustack kernel panic
Hi, This is i386, right ? I think the cause is that assembler routine panic_trigger does not establish the standard i386 frame. Basically, you need either this, or dwarf annotations, for gdb to be able to walk over the frame. You need to add the standard prologue pushl %ebp movl%esp,%ebp and standard epilogue leave to the function. No idea whether it will continue to operate correctly after. my panic_trigger looks like this now: /* int panic_trigger(int *tp) */ ENTRY(panic_trigger) pushl %ebp movl%esp,%ebp xorl%eax, %eax movl$0xdefacedd, %edx lock xchgl %edx, (%edi) cmpl$0, %edx je 0f movl$0, %eax leave ret 0: movl$1, %eax leave ret END(panic_trigger) same result, (actually too same as the address in the stack trace is still the same, is that possible?) cheers --m ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: dtrace ustack kernel panic
Hi, This is i386, right ? I think the cause is that assembler routine panic_trigger does not establish the standard i386 frame. Basically, you need either this, or dwarf annotations, for gdb to be able to walk over the frame. You need to add the standard prologue pushl %ebp movl%esp,%ebp and standard epilogue leave to the function. No idea whether it will continue to operate correctly after. my panic_trigger looks like this now: /* int panic_trigger(int *tp) */ ENTRY(panic_trigger) pushl %ebp movl%esp,%ebp xorl%eax, %eax movl$0xdefacedd, %edx lock xchgl %edx, (%edi) cmpl$0, %edx je 0f movl$0, %eax leave ret 0: movl$1, %eax leave ret END(panic_trigger) same result, (actually too same as the address in the stack trace is still the same, is that possible?) KDB: stack backtrace: #0 0xc09036a7 at kdb_backtrace+0x47 #1 0xc08d1a07 at panic+0x117 #2 0xc0c158c3 at trap_fatal+0x323 #3 0xc0c15bc0 at trap_pfault+0x2f0 #4 0xc0c1612a at trap+0x48a #5 0xc0bfc97c at calltrap+0x6 #6 0xc10e99db at dtrace_panic+0x1b #7 0xc10e9a0d at dtrace_assfail+0x2d #8 0xc10fa6a6 at dtrace_probe+0xfd6 #9 0xc1237ce4 at systrace_probe+0x84 #10 0xc090f63f at syscallenter+0x47f #11 0xc0c15c14 at syscall+0x34 #12 0xc0bfca11 at Xint0x80_syscall+0x21 I tried playing around with kgdb a bit. It finds source files until frame #10 (i.e., syscallenter + 0x47f) (kgdb) list *syscallenter+0x47f 0xc090f63f is in syscallenter (/usr/src/sys/kern/subr_trap.c:328). 323 * If the systrace module has registered it's probe 324 * callback and if there is a probe active for the 325 * syscall 'return', process the probe. 326 */ 327if (systrace_probe_func != NULL sa-callp-sy_return != 0) 328(*systrace_probe_func)(sa-callp-sy_return, sa-code, 329sa-callp, sa-args); 330#endif 331CTR4(KTR_SYSC, syscall: p=%p error=%d return %#lx %#lx, 332p, error, td-td_retval[0], td-td_retval[1]); (kgdb) list *systrace_probe+0x84 No source file for address 0xc1237ce4. Thus, it seems that the first frame where kgdb cannot find a source file is #9 (i.e., systrace_probe + 0x84) As far as I understand that's when the (imho) function pointer systrace_probe_func gets invoked. Therefore, it seems that kgdb has a hard time finding the source file for the function invoked through that pointer. Is this complete nonsense, or does that actually sound familiar to anyone? And still I'm wondering whether anybody ever successfully used the ustack action on Freebsd-8.2 i386 cheers --m ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: dtrace ustack kernel panic
Hi, trying to do so I don't really find my way around. This is what I get when I run kgdb On startup the assert frame is #7 and the probe frame is #8. kgdb kernel.debug /var/crash/vmcore.0 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type show copying to see the conditions. There is absolutely no warranty for GDB. Type show warranty for details. This GDB was configured as i386-marcel-freebsd... Unread portion of the kernel message buffer: kernel trap 12 with interrupts disabled - Show quoted text - Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address= 0x108 fault code= supervisor write, page not present instruction pointer= 0x20:0xc11012d7 stack pointer= 0x28:0xcd3ed9f4 frame pointer= 0x28:0xcd3eda0c code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags= resume, IOPL = 0 current process= 1132 (nc) trap number= 12 panic: page fault cpuid = 0 KDB: stack backtrace: #0 0xc09036a7 at kdb_backtrace+0x47 #1 0xc08d1a07 at panic+0x117 #2 0xc0c158c3 at trap_fatal+0x323 #3 0xc0c15bc0 at trap_pfault+0x2f0 #4 0xc0c1612a at trap+0x48a #5 0xc0bfc97c at calltrap+0x6 #6 0xc10e992b at dtrace_panic+0x1b #7 0xc10e995d at dtrace_assfail+0x2d #8 0xc10fb28c at dtrace_probe+0x135c #9 0xc1237f72 at systrace_probe+0x62 #10 0xc090f63f at syscallenter+0x47f #11 0xc0c15c14 at syscall+0x34 #12 0xc0bfca11 at Xint0x80_syscall+0x21 Uptime: 3m0s Physical memory: 239 MB Dumping 79 MB: 64 48 32 16 Reading symbols from /boot/kernel/dtraceall.ko... Reading symbols from /boot/kernel/dtraceall.ko.symbols...done. done. Loaded symbols for /boot/kernel/dtraceall.ko Reading symbols from /boot/kernel/cyclic.ko...Reading symbols from /boot/kernel/cyclic.ko.symbols...done. done. Loaded symbols for /boot/kernel/cyclic.ko Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /boot/kernel/opensolaris.ko.symbols...done. done. Loaded symbols for /boot/kernel/opensolaris.ko Reading symbols from /boot/kernel/dtrace.ko...Reading symbols from /boot/kernel/dtrace.ko.symbols...done. done. Loaded symbols for /boot/kernel/dtrace.ko Reading symbols from /boot/kernel/dtmalloc.ko...Reading symbols from /boot/kernel/dtmalloc.ko.symbols...done. done. Loaded symbols for /boot/kernel/dtmalloc.ko Reading symbols from /boot/kernel/dtnfsclient.ko...Reading symbols from /boot/kernel/dtnfsclient.ko.symbols...done. done. Loaded symbols for /boot/kernel/dtnfsclient.ko Reading symbols from /boot/kernel/fbt.ko...Reading symbols from /boot/kernel/fbt.ko.symbols...done. done. Loaded symbols for /boot/kernel/fbt.ko Reading symbols from /boot/kernel/lockstat.ko...Reading symbols from /boot/kernel/lockstat.ko.symbols...done. done. Loaded symbols for /boot/kernel/lockstat.ko Reading symbols from /boot/kernel/sdt.ko...Reading symbols from /boot/kernel/sdt.ko.symbols...done. done. Loaded symbols for /boot/kernel/sdt.ko Reading symbols from /boot/kernel/systrace.ko...Reading symbols from /boot/kernel/systrace.ko.symbols...done. done. Loaded symbols for /boot/kernel/systrace.ko Reading symbols from /boot/kernel/profile.ko...Reading symbols from /boot/kernel/profile.ko.symbols...done. done. Loaded symbols for /boot/kernel/profile.ko #0 doadump () at pcpu.h:231 231__asm(movl %%fs:0,%0 : =r (td)); once I'm in the kgdb console i type where and all of a sudden the stack trace has only 7 frames... that does not seem correct. Furthermore, the Previous frame inner to this frame (corrupt stack?) error message is not too encuraging either. (kgdb) where #0 doadump () at pcpu.h:231 #1 0xc08d17a3 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:419 #2 0xc08d1a40 in panic (fmt=Variable fmt is not available. ) at /usr/src/sys/kern/kern_shutdown.c:592 #3 0xc0c158c3 in trap_fatal (frame=0xcd3ed9b4, eva=264) at /usr/src/sys/i386/i386/trap.c:946 #4 0xc0c15bc0 in trap_pfault (frame=0xcd3ed9b4, usermode=0, eva=264) at /usr/src/sys/i386/i386/trap.c:859 #5 0xc0c1612a in trap (frame=0xcd3ed9b4) at /usr/src/sys/i386/i386/trap.c:532 #6 0xc0bfc97c in calltrap () at /usr/src/sys/i386/i386/exception.s:166 #7 0xc11012d7 in dtrace_panic_trigger () from /boot/kernel/dtrace.ko Previous frame inner to this frame (corrupt stack?) (kgdb) what am I doing wrong and what do I have to do to get the specific assert that fails? cheers --m (sorry adriy hI only hit reply the first time not reply-all) On Tue, Jul 26, 2011 at 4:05 AM, Andriy Gapon a...@freebsd.org wrote: on 26/07/2011 04:20 maestro something said the following: Hi, when using the ustack action on the latest FreeBSD8.2 i386 the kernel panics. Here is the information I could gather: let me know if I can provide more information. (i.e., i have the full
Re: dtrace ustack kernel panic
on 26/07/2011 04:20 maestro something said the following: Hi, when using the ustack action on the latest FreeBSD8.2 i386 the kernel panics. Here is the information I could gather: let me know if I can provide more information. (i.e., i have the full crash information 80+mbs handy) Use kgdb on the crash dump, go to the dtrace_probe frame and see which exactly assert fails and why. Here is the backtrace: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x108 fault code = supervisor write, page not present instruction pointer = 0x20:0xc11012d7 stack pointer = 0x28:0xcd3ed9f4 frame pointer = 0x28:0xcd3eda0c code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags= resume, IOPL = 0 current process = 1132 (nc) trap number = 12 panic: page fault cpuid = 0 KDB: stack backtrace: #0 0xc09036a7 at kdb_backtrace+0x47 #1 0xc08d1a07 at panic+0x117 #2 0xc0c158c3 at trap_fatal+0x323 #3 0xc0c15bc0 at trap_pfault+0x2f0 #4 0xc0c1612a at trap+0x48a #5 0xc0bfc97c at calltrap+0x6 #6 0xc10e992b at dtrace_panic+0x1b #7 0xc10e995d at dtrace_assfail+0x2d #8 0xc10fb28c at dtrace_probe+0x135c #9 0xc1237f72 at systrace_probe+0x62 #10 0xc090f63f at syscallenter+0x47f #11 0xc0c15c14 at syscall+0x34 #12 0xc0bfca11 at Xint0x80_syscall+0x21 Uptime: 3m0s Physical memory: 239 MB Dumping 79 MB: 64 48 32 16 Steps To reproduce: the dtrace program (i.e., test.d) was: #!/usr/sbin/dtrace -s syscall::accept:return / execname == nc / { printf(%s accept:return\n, execname); ustack(); } % dtrace -s test.d then running % nc -vl 8080 on one shell and % nc localhost 8080 on another would make the kernel panic -- Andriy Gapon ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org