Re: dtrace ustack kernel panic
Hi, just finished installing FreeBSD-9BETA1 and recompiling the kernel with dtrace. This is even worse, I have the same behavior as mentioned here: http://freebsd.1045724.n5.nabble.com/bin-158431-dtrace-crash-in-dt-proc-lookup-when-attaching-to-PID-assert-dpr-NULL-tt4535367.html#none i.e., dtrace regardless of whether with or without any probes just quits with the following error message Assertion failed: (dpr != NULL), file /usr/src/cddl/lib/libdtrace/../../../cddl/contrib/opensolaris/lib/libdtrace/common/dt_proc.c, line 751. that's inside dt_proc_lookup() I think I have to give up on ustack(), too bad cheers --m On Sat, Jul 30, 2011 at 2:05 PM, maestro something maestr...@gmail.comwrote: Hi, This is i386, right ? I think the cause is that assembler routine panic_trigger does not establish the standard i386 frame. Basically, you need either this, or dwarf annotations, for gdb to be able to walk over the frame. You need to add the standard prologue pushl %ebp movl%esp,%ebp and standard epilogue leave to the function. No idea whether it will continue to operate correctly after. my panic_trigger looks like this now: /* int panic_trigger(int *tp) */ ENTRY(panic_trigger) pushl %ebp movl%esp,%ebp xorl%eax, %eax movl$0xdefacedd, %edx lock xchgl %edx, (%edi) cmpl$0, %edx je 0f movl$0, %eax leave ret 0: movl$1, %eax leave ret END(panic_trigger) same result, (actually too same as the address in the stack trace is still the same, is that possible?) KDB: stack backtrace: #0 0xc09036a7 at kdb_backtrace+0x47 #1 0xc08d1a07 at panic+0x117 #2 0xc0c158c3 at trap_fatal+0x323 #3 0xc0c15bc0 at trap_pfault+0x2f0 #4 0xc0c1612a at trap+0x48a #5 0xc0bfc97c at calltrap+0x6 #6 0xc10e99db at dtrace_panic+0x1b #7 0xc10e9a0d at dtrace_assfail+0x2d #8 0xc10fa6a6 at dtrace_probe+0xfd6 #9 0xc1237ce4 at systrace_probe+0x84 #10 0xc090f63f at syscallenter+0x47f #11 0xc0c15c14 at syscall+0x34 #12 0xc0bfca11 at Xint0x80_syscall+0x21 I tried playing around with kgdb a bit. It finds source files until frame #10 (i.e., syscallenter + 0x47f) (kgdb) list *syscallenter+0x47f 0xc090f63f is in syscallenter (/usr/src/sys/kern/subr_trap.c:328). 323 * If the systrace module has registered it's probe 324 * callback and if there is a probe active for the 325 * syscall 'return', process the probe. 326 */ 327if (systrace_probe_func != NULL sa-callp-sy_return != 0) 328(*systrace_probe_func)(sa-callp-sy_return, sa-code, 329sa-callp, sa-args); 330#endif 331CTR4(KTR_SYSC, syscall: p=%p error=%d return %#lx %#lx, 332p, error, td-td_retval[0], td-td_retval[1]); (kgdb) list *systrace_probe+0x84 No source file for address 0xc1237ce4. Thus, it seems that the first frame where kgdb cannot find a source file is #9 (i.e., systrace_probe + 0x84) As far as I understand that's when the (imho) function pointer systrace_probe_func gets invoked. Therefore, it seems that kgdb has a hard time finding the source file for the function invoked through that pointer. Is this complete nonsense, or does that actually sound familiar to anyone? And still I'm wondering whether anybody ever successfully used the ustack action on Freebsd-8.2 i386 cheers --m ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: dtrace ustack kernel panic
Hi,
This is even worse, I have the same behavior as mentioned here:
http://freebsd.1045724.n5.nabble.com/bin-158431-dtrace-crash-in-dt-proc-lookup-when-attaching-to-PID-assert-dpr-NULL-tt4535367.html#none
i.e., dtrace regardless of whether with or without any probes just quits
with the following error message
this is not accurate. It seems that this message only occurs if i use the
ustack() action.
i.e,
dtrace -n 'syscall::accept:return /execname == nc / { printf(%s
accept:return\n, execname);}'
works as expected
However,
dtrace -n 'syscall::accept:return /execname == nc / { printf(%s
accept:return\n, execname);ustack();}'
does not:
it results in the mentioned error message.
Assertion failed: (dpr != NULL), file
/usr/src/cddl/lib/libdtrace/../../../cddl/contrib/opensolaris/lib/libdtrace/common/dt_proc.c,
line 751.
cheers
--m
Assertion failed: (dpr != NULL), file
/usr/src/cddl/lib/libdtrace/../../../cddl/contrib/opensolaris/lib/libdtrace/common/dt_proc.c,
line 751.
that's inside dt_proc_lookup()
I think I have to give up on ustack(), too bad
cheers
--m
On Sat, Jul 30, 2011 at 2:05 PM, maestro something maestr...@gmail.comwrote:
Hi,
This is i386, right ?
I think the cause is that assembler routine panic_trigger does not
establish the standard i386 frame. Basically, you need either this,
or dwarf annotations, for gdb to be able to walk over the frame.
You need to add the standard prologue
pushl %ebp
movl%esp,%ebp
and standard epilogue
leave
to the function. No idea whether it will continue to operate correctly
after.
my panic_trigger looks like this now:
/*
int
panic_trigger(int *tp)
*/
ENTRY(panic_trigger)
pushl %ebp
movl%esp,%ebp
xorl%eax, %eax
movl$0xdefacedd, %edx
lock
xchgl %edx, (%edi)
cmpl$0, %edx
je 0f
movl$0, %eax
leave
ret
0: movl$1, %eax
leave
ret
END(panic_trigger)
same result, (actually too same as the address in the stack trace is
still the same, is that possible?)
KDB: stack backtrace:
#0 0xc09036a7 at kdb_backtrace+0x47
#1 0xc08d1a07 at panic+0x117
#2 0xc0c158c3 at trap_fatal+0x323
#3 0xc0c15bc0 at trap_pfault+0x2f0
#4 0xc0c1612a at trap+0x48a
#5 0xc0bfc97c at calltrap+0x6
#6 0xc10e99db at dtrace_panic+0x1b
#7 0xc10e9a0d at dtrace_assfail+0x2d
#8 0xc10fa6a6 at dtrace_probe+0xfd6
#9 0xc1237ce4 at systrace_probe+0x84
#10 0xc090f63f at syscallenter+0x47f
#11 0xc0c15c14 at syscall+0x34
#12 0xc0bfca11 at Xint0x80_syscall+0x21
I tried playing around with kgdb a bit. It finds source files until frame
#10 (i.e., syscallenter + 0x47f)
(kgdb) list *syscallenter+0x47f
0xc090f63f is in syscallenter (/usr/src/sys/kern/subr_trap.c:328).
323 * If the systrace module has registered it's probe
324 * callback and if there is a probe active for the
325 * syscall 'return', process the probe.
326 */
327if (systrace_probe_func != NULL sa-callp-sy_return !=
0)
328(*systrace_probe_func)(sa-callp-sy_return, sa-code,
329sa-callp, sa-args);
330#endif
331CTR4(KTR_SYSC, syscall: p=%p error=%d return %#lx %#lx,
332p, error, td-td_retval[0], td-td_retval[1]);
(kgdb) list *systrace_probe+0x84
No source file for address 0xc1237ce4.
Thus, it seems that the first frame where kgdb cannot find a source file
is #9 (i.e., systrace_probe + 0x84)
As far as I understand that's when the (imho) function pointer
systrace_probe_func gets invoked. Therefore, it seems that kgdb has a hard
time finding the source file for the function invoked through that pointer.
Is this complete nonsense, or does that actually sound familiar to anyone?
And still I'm wondering whether anybody ever successfully used the ustack
action on Freebsd-8.2 i386
cheers
--m
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: dtrace ustack kernel panic
Hi,
http://freebsd.1045724.n5.nabble.com/bin-158431-dtrace-crash-in-dt-proc-lookup-when-attaching-to-PID-assert-dpr-NULL-tt4535367.html#none
Kind of a mentoring note: it would be much shorter and much more useful to
paste
PR 158431 or, even better, an http URL to the said PR in FreeBSD PR DB
web
interface. I know, I know, it's really Google to blame, right? :-)
True, google was showing this thread as a result to my search query.
i.e., dtrace regardless of whether with or without any probes just quits
with
the following error message
Assertion failed: (dpr != NULL), file
/usr/src/cddl/lib/libdtrace/../../../cddl/contrib/opensolaris/lib/libdtrace/common/dt_proc.c,
line 751.
that's inside dt_proc_lookup()
I think I have to give up on ustack(), too bad
Or maybe you still have a chance to help us here, maybe it will even help
you in
the long term.
It would be good if you pasted a little bit more of the output that you
got.
Please note that in that PR the (meaningful) output starts with:
dtrace: no probes specified
I.e. it starts with what dtrace considers to be a fatal condition.
I tried to run dtruss (as you did) and I got this pre-amble before the
assertion:
[some dtrace script body]
: probe description proc:::exit does not match any probes
ok this is the command line and all the output i get from dtrace:
fb90i386# dtrace -n 'syscall::accept:return / execname == nc / {
@sc[execname,probefunc] = count(); printf(%s accept:return\n, execname);
ustack();}'
dtrace: description 'syscall::accept:return ' matched 1 probe
CPU IDFUNCTION:NAME
0 46457accept:return nc accept:return
Assertion failed: (dpr != NULL), file
/usr/src/cddl/lib/libdtrace/../../../cddl/contrib/opensolaris/lib/libdtrace/common/dt_proc.c,
line 751.
Abort (core dumped)
fb90i386#
I get the same thing when using dtruss:
fb90i386# dtruss -a -p 1139
Assertion failed: (dpr != NULL), file
/usr/src/cddl/lib/libdtrace/../../../cddl/contrib/opensolaris/lib/libdtrace/common/dt_proc.c,
line 751.
Abort trap (core dumped)
fb90i386#
I guess that in my case I got it because my userland was not compiled with
CTF
support. Not sure about yours...
hm userland and CTF not that I know. I only recompiled the kernel with CTF.
Do I have to recompile userland (and would that be make world?) If so can i
just pass the same WITH_CTF=1 option to make?
Of course, it's still rather bad that dtrace crashes when it prematurely
exits.
But maybe it doesn't crash in the correct environment... I don't know.
as soon as I understand what you mean by userland I can recompile the
necessary bits.
cheers
and thanks a lot for your patience
--m
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: dtrace ustack kernel panic
Hi, diff --git a/lib/libproc/proc_create.c b/lib/libproc/proc_create.c index c372a47..9bd24a2 100644 --- a/lib/libproc/proc_create.c +++ b/lib/libproc/proc_create.c @@ -79,12 +79,11 @@ proc_attach(pid_t pid, int flags, struct proc_handle **pphdl) else phdl-status = PS_STOP; +out: if (error) proc_free(phdl); else *pphdl = phdl; -out: - proc_free(phdl); return (error); } What do I have to recompile for the patch to be picked up? make libproc in /usr/src/lib works but make install libproc fails the following way: fb90i386# make libproc Warning: Object directory not changed from original /usr/src/lib/libproc cc -O2 -pipe -I/usr/src/lib/libproc -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -c proc_bkpt.c cc -O2 -pipe -I/usr/src/lib/libproc -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -c proc_create.c cc -O2 -pipe -I/usr/src/lib/libproc -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -c proc_regs.c cc -O2 -pipe -I/usr/src/lib/libproc -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -c proc_sym.c cc -O2 -pipe -I/usr/src/lib/libproc -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -c proc_rtld.c cc -O2 -pipe -I/usr/src/lib/libproc -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -c proc_util.c building static proc library ranlib libproc.a cc -pg -O2 -pipe -I/usr/src/lib/libproc -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -c proc_bkpt.c -o proc_bkpt.po cc -pg -O2 -pipe -I/usr/src/lib/libproc -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -c proc_create.c -o proc_create.po cc -pg -O2 -pipe -I/usr/src/lib/libproc -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -c proc_regs.c -o proc_regs.po cc -pg -O2 -pipe -I/usr/src/lib/libproc -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -c proc_sym.c -o proc_sym.po cc -pg -O2 -pipe -I/usr/src/lib/libproc -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall
Re: dtrace ustack kernel panic
Hi,
Hi,
diff --git a/lib/libproc/proc_create.c b/lib/libproc/proc_create.c
index c372a47..9bd24a2 100644
--- a/lib/libproc/proc_create.c
+++ b/lib/libproc/proc_create.c
@@ -79,12 +79,11 @@ proc_attach(pid_t pid, int flags, struct proc_handle
**pphdl)
else
phdl-status = PS_STOP;
+out:
if (error)
proc_free(phdl);
else
*pphdl = phdl;
-out:
- proc_free(phdl);
return (error);
}
What do I have to recompile for the patch to be picked up?
make libproc in /usr/src/lib
works but
make install libproc
fails the following way:
fb90i386# make install libproc
=== csu/i386-elf (install)
install -o root -g wheel -m 444 crti.o crtn.o gcrt1.o crt1.o Scrt1.o
/usr/lib
=== libc (install)
install -C -o root -g wheel -m 444 libc.a /usr/lib
install: libc.a: No such file or directory
*** Error code 71
Stop in /usr/src/lib/libc.
*** Error code 1
Stop in /usr/src/lib.
Did you remember to 'make obj'? If you didn't, the library would be
left in the wrong place where 'make install' would not find it.
make obj
make depend (possibly a no-op)
make
make install
Thank you very much that helped.
now it seems that the ustack() action on i386 and FreeBSD-9.0BETA1 almost
works.
Almost because dtrace still exits (this time without an error message) The
process for which the ustack should be genereated is terminated
Here are the specifics:
fb90i386# dtrace -n 'syscall::accept:return / execname == nc / {
ustack();}'
dtrace: description 'syscall::accept:return ' matched 1 probe
CPU IDFUNCTION:NAME
0 46457accept:return
libc.so.7`__sys_accept+0x7
0x3
0xed96e824
This alsmost looks like a stack trace I'd say
However, dtrace quits once I connect to nc
nc quits with the following error message
fb90i386# nc -vl
nc: Polling Error: Interrupted system call
cheers
-m
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: dtrace ustack kernel panic
Running the same example on amd64 has the same behavior as on i386 (granted the backtrace is slightly different). That is, the kernel panics immediately when you use the ustack() action. Furthermore, the crashdump does not provide the nececcary information to get at the root of the cause (i.e., no souce found for the systrace_probe and dtrace_probe frames). still I'm wondering ... Did anybody successfully use the ustack() action on FreeBSD-8.2? cheers --m backtrace on amd64: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address= 0x0 fault code= supervisor read data, page not present instruction pointer= 0x20:0x80ff1fa6 stack pointer= 0x28:0xff80001d7500 frame pointer= 0x28:0xff80001d7850 code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags= resume, IOPL = 0 current process= 1068 (nc) trap number= 12 panic: page fault cpuid = 0 KDB: stack backtrace: #0 0x80618d2e at kdb_backtrace+0x5e #1 0x805e4d37 at panic+0x187 #2 0x808dcac0 at trap_fatal+0x290 #3 0x808dcf0d at trap_pfault+0x2fd #4 0x808dd343 at trap+0x343 #5 0x808c5298 at calltrap+0x8 #6 0x80ff638a at dtrace_probe+0x47a #7 0x81192285 at systrace_probe+0x45 #8 0x80624f50 at syscallenter+0x3c0 #9 0x808dcf7b at syscall+0x4b #10 0x808c5572 at Xfast_syscall+0xe2 Uptime: 3m0s On Sun, Jul 31, 2011 at 6:46 AM, Alex Samorukov m...@os2.kiev.ua wrote: Hi, I just tried your test on -STABLE + 1 fix from current [1] and got no trap. dtrace: buffer size lowered to 2m CPU IDFUNCTION:NAME 4 42224accept:return nc accept:return Assertion failed: (dpr != NULL), file /usr/src/cddl/lib/libdtrace/..** /../../cddl/contrib/**opensolaris/lib/libdtrace/**common/dt_proc.c, line 751. Abort (core dumped) Also 1st dtrace was killed as well. I tried 2 times with the same result. Box is amd64. This is bt from gdb: (gdb) bt #0 0x000800fa2f9c in thr_kill () from /lib/libc.so.7 #1 0x00080103f6cb in abort () from /lib/libc.so.7 #2 0x000800786758 in dt_proc_lookup () from /lib/libdtrace.so.2 #3 0x0008007867db in dt_proc_lock () from /lib/libdtrace.so.2 #4 0x0008007a5bcb in dt_print_ustack () from /lib/libdtrace.so.2 #5 0x0008007a8504 in dt_print_agg () from /lib/libdtrace.so.2 #6 0x0008007a88ca in dtrace_consume () from /lib/libdtrace.so.2 #7 0x00080077e1a2 in dtrace_work () from /lib/libdtrace.so.2 #8 0x004044ba in ?? () #9 0x0040222e in ?? () [skip] [1] http://www.freebsd.org/cgi/**query-pr.cgi?pr=kern/159064http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/159064 On 07/26/2011 03:20 AM, maestro something wrote: Hi, when using the ustack action on the latest FreeBSD8.2 i386 the kernel panics. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: dtrace ustack kernel panic
On Sat, Jul 30, 2011 at 12:50 AM, Andriy Gapon a...@freebsd.org wrote:
on 30/07/2011 00:27 maestro something said the following:
Hi,
trying to do so I don't really find my way around. This is what I get
when I run
kgdb
On startup the assert frame is #7 and the probe frame is #8.
[snip]
kernel trap 12 with interrupts disabled
I am not quite sure. Apparently there is some issue with either what
information we store in a crash dump and how, or with how kgdb interprets
that
information, or with a combination of both...
Perhaps this is a result of -O2 optimization and -O1 would improve the
situation
I guess I have to recompile the kernel without -O then ... does that work
for freebsd (I remember linux has issues i.e., does not compile without any
-O)
How would I do that the in the most straight forward way?
Meanwhile, there is something simple that you can do without much hassle:
(kgdb) list *dtrace_probe+0x135c
True there was not a lot hassle involved. However the result is not really
satisfactory either :-)
(kgdb) list *dtrace_probe+0x135c
No source file for address 0xc10fb28c.
The address is in accordance with the stack-trace (i.e., frame #8)
where the address is taken from the backtrace printed by panic(9).
cheers
--m
On Tue, Jul 26, 2011 at 4:05 AM, Andriy Gapon a...@freebsd.org
mailto:a...@freebsd.org wrote:
on 26/07/2011 04:20 maestro something said the following:
Hi,
when using the ustack action on the latest FreeBSD8.2 i386 the
kernel
panics.
Here is the information I could gather:
let me know if I can provide more information. (i.e., i have the
full crash
information 80+mbs handy)
Use kgdb on the crash dump, go to the dtrace_probe frame and see
which exactly
assert fails and why.
Here is the backtrace:
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x108
fault code = supervisor write, page not present
instruction pointer = 0x20:0xc11012d7
stack pointer = 0x28:0xcd3ed9f4
frame pointer = 0x28:0xcd3eda0c
code segment= base 0x0, limit 0xf, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags= resume, IOPL = 0
current process = 1132 (nc)
trap number = 12
panic: page fault
cpuid = 0
KDB: stack backtrace:
#0 0xc09036a7 at kdb_backtrace+0x47
#1 0xc08d1a07 at panic+0x117
#2 0xc0c158c3 at trap_fatal+0x323
#3 0xc0c15bc0 at trap_pfault+0x2f0
#4 0xc0c1612a at trap+0x48a
#5 0xc0bfc97c at calltrap+0x6
#6 0xc10e992b at dtrace_panic+0x1b
#7 0xc10e995d at dtrace_assfail+0x2d
#8 0xc10fb28c at dtrace_probe+0x135c
#9 0xc1237f72 at systrace_probe+0x62
#10 0xc090f63f at syscallenter+0x47f
#11 0xc0c15c14 at syscall+0x34
#12 0xc0bfca11 at Xint0x80_syscall+0x21
Uptime: 3m0s
Physical memory: 239 MB
Dumping 79 MB: 64 48 32 16
Steps To reproduce:
the dtrace program (i.e., test.d) was:
#!/usr/sbin/dtrace -s
syscall::accept:return
/ execname == nc /
{
printf(%s accept:return\n, execname);
ustack();
}
% dtrace -s test.d
then running
% nc -vl 8080
on one shell and
% nc localhost 8080
on another would make the kernel panic
--
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: dtrace ustack kernel panic
On Sat, Jul 30, 2011 at 11:27 AM, Andriy Gapon a...@freebsd.org wrote:
on 30/07/2011 21:19 maestro something said the following:
On Sat, Jul 30, 2011 at 12:50 AM, Andriy Gapon a...@freebsd.org
mailto:a...@freebsd.org wrote:
on 30/07/2011 00:27 maestro something said the following:
Hi,
trying to do so I don't really find my way around. This is what I
get when
I run
kgdb
On startup the assert frame is #7 and the probe frame is #8.
[snip]
kernel trap 12 with interrupts disabled
I am not quite sure. Apparently there is some issue with either what
information we store in a crash dump and how, or with how kgdb
interprets that
information, or with a combination of both...
Perhaps this is a result of -O2 optimization and -O1 would improve
the situation
I guess I have to recompile the kernel without -O then ... does that work
for
freebsd (I remember linux has issues i.e., does not compile without any
-O)
Not sure about -O0; -O1/-O should work fine.
How would I do that the in the most straight forward way?
See man make.conf: CFLAGS, etc.
Looking into that as I type.
Meanwhile, there is something simple that you can do without much
hassle:
(kgdb) list *dtrace_probe+0x135c
True there was not a lot hassle involved. However the result is not
really
satisfactory either :-)
(kgdb) list *dtrace_probe+0x135c
No source file for address 0xc10fb28c.
The address is in accordance with the stack-trace (i.e., frame #8)
where the address is taken from the backtrace printed by panic(9).
Have you started kgdb with the correct kernel and core file?
If yes, then I am out of ideas.
I hope so, I only recompiled the kernel once according to the DTRACE wiki
instructions and I certainly only have one /var/crash/vmcore.* file.
I'll try recompiling the kernel with -O1 and try again. In the meantime, I'm
wondering whether I'm really the only/first one that ran into this problem
or if there are people that actually successfully used the ustack() target
on freebsd-8.2?
cheers
--m
On Tue, Jul 26, 2011 at 4:05 AM, Andriy Gapon a...@freebsd.org
mailto:a...@freebsd.org
mailto:a...@freebsd.org mailto:a...@freebsd.org wrote:
on 26/07/2011 04:20 maestro something said the following:
Hi,
when using the ustack action on the latest FreeBSD8.2 i386
the kernel
panics.
Here is the information I could gather:
let me know if I can provide more information. (i.e., i have
the
full crash
information 80+mbs handy)
Use kgdb on the crash dump, go to the dtrace_probe frame and
see which
exactly
assert fails and why.
Here is the backtrace:
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x108
fault code = supervisor write, page not present
instruction pointer = 0x20:0xc11012d7
stack pointer = 0x28:0xcd3ed9f4
frame pointer = 0x28:0xcd3eda0c
code segment= base 0x0, limit 0xf, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags= resume, IOPL = 0
current process = 1132 (nc)
trap number = 12
panic: page fault
cpuid = 0
KDB: stack backtrace:
#0 0xc09036a7 at kdb_backtrace+0x47
#1 0xc08d1a07 at panic+0x117
#2 0xc0c158c3 at trap_fatal+0x323
#3 0xc0c15bc0 at trap_pfault+0x2f0
#4 0xc0c1612a at trap+0x48a
#5 0xc0bfc97c at calltrap+0x6
#6 0xc10e992b at dtrace_panic+0x1b
#7 0xc10e995d at dtrace_assfail+0x2d
#8 0xc10fb28c at dtrace_probe+0x135c
#9 0xc1237f72 at systrace_probe+0x62
#10 0xc090f63f at syscallenter+0x47f
#11 0xc0c15c14 at syscall+0x34
#12 0xc0bfca11 at Xint0x80_syscall+0x21
Uptime: 3m0s
Physical memory: 239 MB
Dumping 79 MB: 64 48 32 16
Steps To reproduce:
the dtrace program (i.e., test.d) was:
#!/usr/sbin/dtrace -s
syscall::accept:return
/ execname == nc /
{
printf(%s accept:return\n, execname);
ustack();
}
% dtrace -s test.d
then running
% nc -vl 8080
on one shell and
% nc localhost 8080
on another would make the kernel panic
--
Andriy Gapon
Re: dtrace ustack kernel panic
Hi, Have you started kgdb with the correct kernel and core file? If yes, then I am out of ideas. I hope so, I only recompiled the kernel once according to the DTRACE wiki instructions and I certainly only have one /var/crash/vmcore.* file. I'll try recompiling the kernel with -O1 and try again. In the meantime, I'm wondering whether I'm really the only/first one that ran into this problem or if there are people that actually successfully used the ustack() target on freebsd-8.2? I could not get the information even after recompiling the kernel here is the relevant (I think information). fb82i386# cat /etc/make.conf CFLAGS= -O (accodring to man make.conf only -O and -O2 is supported for CFLAGS anyways) kernel.debug is the newly compiled kernel (according to the timestamp) fb82i386# kgdb kernel.debug /var/crash/vmcore.0 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type show copying to see the conditions. There is absolutely no warranty for GDB. Type show warranty for details. This GDB was configured as i386-marcel-freebsd... Unread portion of the kernel message buffer: kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address= 0x108 fault code= supervisor write, page not present instruction pointer= 0x20:0xc1100847 stack pointer= 0x28:0xcd39a9e4 frame pointer= 0x28:0xcd39a9fc code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags= resume, IOPL = 0 current process= 1060 (nc) trap number= 12 panic: page fault cpuid = 0 KDB: stack backtrace: #0 0xc09036a7 at kdb_backtrace+0x47 #1 0xc08d1a07 at panic+0x117 #2 0xc0c158c3 at trap_fatal+0x323 #3 0xc0c15bc0 at trap_pfault+0x2f0 #4 0xc0c1612a at trap+0x48a #5 0xc0bfc97c at calltrap+0x6 #6 0xc10e99db at dtrace_panic+0x1b #7 0xc10e9a0d at dtrace_assfail+0x2d #8 0xc10fa6a6 at dtrace_probe+0xfd6 #9 0xc1237ce4 at systrace_probe+0x84 #10 0xc090f63f at syscallenter+0x47f #11 0xc0c15c14 at syscall+0x34 #12 0xc0bfca11 at Xint0x80_syscall+0x21 Uptime: 2m39s Physical memory: 239 MB Dumping 78 MB: 63 47 31 15 (kgdb) where #0 doadump () at pcpu.h:231 #1 0xc08d17a3 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:419 #2 0xc08d1a40 in panic (fmt=Variable fmt is not available. ) at /usr/src/sys/kern/kern_shutdown.c:592 #3 0xc0c158c3 in trap_fatal (frame=0xcd39a9a4, eva=264) at /usr/src/sys/i386/i386/trap.c:946 #4 0xc0c15bc0 in trap_pfault (frame=0xcd39a9a4, usermode=0, eva=264) at /usr/src/sys/i386/i386/trap.c:859 #5 0xc0c1612a in trap (frame=0xcd39a9a4) at /usr/src/sys/i386/i386/trap.c:532 #6 0xc0bfc97c in calltrap () at /usr/src/sys/i386/i386/exception.s:166 #7 0xc1100847 in dtrace_panic_trigger () from /boot/kernel/dtrace.ko Previous frame inner to this frame (corrupt stack?) (kgdb) list *dtrace_probe+0xfd6 No source file for address 0xc10fa6a6. So I'm stuck at the same point. any other ideas? cheers --m cheers --m On Tue, Jul 26, 2011 at 4:05 AM, Andriy Gapon a...@freebsd.org mailto:a...@freebsd.org mailto:a...@freebsd.org mailto:a...@freebsd.org wrote: on 26/07/2011 04:20 maestro something said the following: Hi, when using the ustack action on the latest FreeBSD8.2 i386 the kernel panics. Here is the information I could gather: let me know if I can provide more information. (i.e., i have the full crash information 80+mbs handy) Use kgdb on the crash dump, go to the dtrace_probe frame and see which exactly assert fails and why. Here is the backtrace: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x108 fault code = supervisor write, page not present instruction pointer = 0x20:0xc11012d7 stack pointer = 0x28:0xcd3ed9f4 frame pointer = 0x28:0xcd3eda0c code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags= resume, IOPL = 0 current process = 1132 (nc) trap number = 12 panic: page fault cpuid = 0 KDB: stack backtrace: #0 0xc09036a7 at kdb_backtrace+0x47 #1 0xc08d1a07 at panic+0x117 #2 0xc0c158c3 at trap_fatal+0x323 #3 0xc0c15bc0 at trap_pfault+0x2f0 #4 0xc0c1612a at trap+0x48a #5 0xc0bfc97c
Re: dtrace ustack kernel panic
Hi,
On Sat, Jul 30, 2011 at 12:52 PM, Andriy Gapon a...@freebsd.org wrote:
on 30/07/2011 22:05 maestro something said the following:
fb82i386# cat /etc/make.conf
CFLAGS= -O
(accodring to man make.conf only -O and -O2 is supported for CFLAGS
anyways)
kernel.debug is the newly compiled kernel (according to the timestamp)
fb82i386# kgdb kernel.debug /var/crash/vmcore.0
Is vmcore.0 also new (i.e. produced with the new kernel)?
Does 'list ...' command still produces no useful output?
As mentioned before, the list command still gives the same error message:
P.S. Also, hmm, I think that you shouldn't run kgdb on kernel.debug, just
to
avoid any confusion... I think that you should run something like kgdb
/boot/${your installed kernel}/kernel /var/crash/vmcore.0; the debug
symbols are
automatically picked up from *.symbols files in the kernel directory.
Same result when using /boot/kernel/kernel
(kgdb) list *dtrace_probe+0xfd6
No source file for address 0xc10fa6a6.
cheers
--m
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: dtrace ustack kernel panic
Hi, on 30/07/2011 23:03 maestro something said the following: (kgdb) list *dtrace_probe+0xfd6 No source file for address 0xc10fa6a6. Are the sources on the same machine? This is probably the last idea from me. Yes all is done on the same (virtual 32bit) machine. Sources are there too. If i do make installkernel KERNCONF=MYKERNEL the kernel should be installed right? (I assume so b/c /boot/kernel/kernel has the correct new timestamp) cheers -m ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: dtrace ustack kernel panic
Hi, This is i386, right ? I think the cause is that assembler routine panic_trigger does not establish the standard i386 frame. Basically, you need either this, or dwarf annotations, for gdb to be able to walk over the frame. You need to add the standard prologue pushl %ebp movl%esp,%ebp and standard epilogue leave to the function. No idea whether it will continue to operate correctly after. my panic_trigger looks like this now: /* int panic_trigger(int *tp) */ ENTRY(panic_trigger) pushl %ebp movl%esp,%ebp xorl%eax, %eax movl$0xdefacedd, %edx lock xchgl %edx, (%edi) cmpl$0, %edx je 0f movl$0, %eax leave ret 0: movl$1, %eax leave ret END(panic_trigger) same result, (actually too same as the address in the stack trace is still the same, is that possible?) cheers --m ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: dtrace ustack kernel panic
Hi, This is i386, right ? I think the cause is that assembler routine panic_trigger does not establish the standard i386 frame. Basically, you need either this, or dwarf annotations, for gdb to be able to walk over the frame. You need to add the standard prologue pushl %ebp movl%esp,%ebp and standard epilogue leave to the function. No idea whether it will continue to operate correctly after. my panic_trigger looks like this now: /* int panic_trigger(int *tp) */ ENTRY(panic_trigger) pushl %ebp movl%esp,%ebp xorl%eax, %eax movl$0xdefacedd, %edx lock xchgl %edx, (%edi) cmpl$0, %edx je 0f movl$0, %eax leave ret 0: movl$1, %eax leave ret END(panic_trigger) same result, (actually too same as the address in the stack trace is still the same, is that possible?) KDB: stack backtrace: #0 0xc09036a7 at kdb_backtrace+0x47 #1 0xc08d1a07 at panic+0x117 #2 0xc0c158c3 at trap_fatal+0x323 #3 0xc0c15bc0 at trap_pfault+0x2f0 #4 0xc0c1612a at trap+0x48a #5 0xc0bfc97c at calltrap+0x6 #6 0xc10e99db at dtrace_panic+0x1b #7 0xc10e9a0d at dtrace_assfail+0x2d #8 0xc10fa6a6 at dtrace_probe+0xfd6 #9 0xc1237ce4 at systrace_probe+0x84 #10 0xc090f63f at syscallenter+0x47f #11 0xc0c15c14 at syscall+0x34 #12 0xc0bfca11 at Xint0x80_syscall+0x21 I tried playing around with kgdb a bit. It finds source files until frame #10 (i.e., syscallenter + 0x47f) (kgdb) list *syscallenter+0x47f 0xc090f63f is in syscallenter (/usr/src/sys/kern/subr_trap.c:328). 323 * If the systrace module has registered it's probe 324 * callback and if there is a probe active for the 325 * syscall 'return', process the probe. 326 */ 327if (systrace_probe_func != NULL sa-callp-sy_return != 0) 328(*systrace_probe_func)(sa-callp-sy_return, sa-code, 329sa-callp, sa-args); 330#endif 331CTR4(KTR_SYSC, syscall: p=%p error=%d return %#lx %#lx, 332p, error, td-td_retval[0], td-td_retval[1]); (kgdb) list *systrace_probe+0x84 No source file for address 0xc1237ce4. Thus, it seems that the first frame where kgdb cannot find a source file is #9 (i.e., systrace_probe + 0x84) As far as I understand that's when the (imho) function pointer systrace_probe_func gets invoked. Therefore, it seems that kgdb has a hard time finding the source file for the function invoked through that pointer. Is this complete nonsense, or does that actually sound familiar to anyone? And still I'm wondering whether anybody ever successfully used the ustack action on Freebsd-8.2 i386 cheers --m ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: dtrace ustack kernel panic
Hi, trying to do so I don't really find my way around. This is what I get when I run kgdb On startup the assert frame is #7 and the probe frame is #8. kgdb kernel.debug /var/crash/vmcore.0 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type show copying to see the conditions. There is absolutely no warranty for GDB. Type show warranty for details. This GDB was configured as i386-marcel-freebsd... Unread portion of the kernel message buffer: kernel trap 12 with interrupts disabled - Show quoted text - Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address= 0x108 fault code= supervisor write, page not present instruction pointer= 0x20:0xc11012d7 stack pointer= 0x28:0xcd3ed9f4 frame pointer= 0x28:0xcd3eda0c code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags= resume, IOPL = 0 current process= 1132 (nc) trap number= 12 panic: page fault cpuid = 0 KDB: stack backtrace: #0 0xc09036a7 at kdb_backtrace+0x47 #1 0xc08d1a07 at panic+0x117 #2 0xc0c158c3 at trap_fatal+0x323 #3 0xc0c15bc0 at trap_pfault+0x2f0 #4 0xc0c1612a at trap+0x48a #5 0xc0bfc97c at calltrap+0x6 #6 0xc10e992b at dtrace_panic+0x1b #7 0xc10e995d at dtrace_assfail+0x2d #8 0xc10fb28c at dtrace_probe+0x135c #9 0xc1237f72 at systrace_probe+0x62 #10 0xc090f63f at syscallenter+0x47f #11 0xc0c15c14 at syscall+0x34 #12 0xc0bfca11 at Xint0x80_syscall+0x21 Uptime: 3m0s Physical memory: 239 MB Dumping 79 MB: 64 48 32 16 Reading symbols from /boot/kernel/dtraceall.ko... Reading symbols from /boot/kernel/dtraceall.ko.symbols...done. done. Loaded symbols for /boot/kernel/dtraceall.ko Reading symbols from /boot/kernel/cyclic.ko...Reading symbols from /boot/kernel/cyclic.ko.symbols...done. done. Loaded symbols for /boot/kernel/cyclic.ko Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /boot/kernel/opensolaris.ko.symbols...done. done. Loaded symbols for /boot/kernel/opensolaris.ko Reading symbols from /boot/kernel/dtrace.ko...Reading symbols from /boot/kernel/dtrace.ko.symbols...done. done. Loaded symbols for /boot/kernel/dtrace.ko Reading symbols from /boot/kernel/dtmalloc.ko...Reading symbols from /boot/kernel/dtmalloc.ko.symbols...done. done. Loaded symbols for /boot/kernel/dtmalloc.ko Reading symbols from /boot/kernel/dtnfsclient.ko...Reading symbols from /boot/kernel/dtnfsclient.ko.symbols...done. done. Loaded symbols for /boot/kernel/dtnfsclient.ko Reading symbols from /boot/kernel/fbt.ko...Reading symbols from /boot/kernel/fbt.ko.symbols...done. done. Loaded symbols for /boot/kernel/fbt.ko Reading symbols from /boot/kernel/lockstat.ko...Reading symbols from /boot/kernel/lockstat.ko.symbols...done. done. Loaded symbols for /boot/kernel/lockstat.ko Reading symbols from /boot/kernel/sdt.ko...Reading symbols from /boot/kernel/sdt.ko.symbols...done. done. Loaded symbols for /boot/kernel/sdt.ko Reading symbols from /boot/kernel/systrace.ko...Reading symbols from /boot/kernel/systrace.ko.symbols...done. done. Loaded symbols for /boot/kernel/systrace.ko Reading symbols from /boot/kernel/profile.ko...Reading symbols from /boot/kernel/profile.ko.symbols...done. done. Loaded symbols for /boot/kernel/profile.ko #0 doadump () at pcpu.h:231 231__asm(movl %%fs:0,%0 : =r (td)); once I'm in the kgdb console i type where and all of a sudden the stack trace has only 7 frames... that does not seem correct. Furthermore, the Previous frame inner to this frame (corrupt stack?) error message is not too encuraging either. (kgdb) where #0 doadump () at pcpu.h:231 #1 0xc08d17a3 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:419 #2 0xc08d1a40 in panic (fmt=Variable fmt is not available. ) at /usr/src/sys/kern/kern_shutdown.c:592 #3 0xc0c158c3 in trap_fatal (frame=0xcd3ed9b4, eva=264) at /usr/src/sys/i386/i386/trap.c:946 #4 0xc0c15bc0 in trap_pfault (frame=0xcd3ed9b4, usermode=0, eva=264) at /usr/src/sys/i386/i386/trap.c:859 #5 0xc0c1612a in trap (frame=0xcd3ed9b4) at /usr/src/sys/i386/i386/trap.c:532 #6 0xc0bfc97c in calltrap () at /usr/src/sys/i386/i386/exception.s:166 #7 0xc11012d7 in dtrace_panic_trigger () from /boot/kernel/dtrace.ko Previous frame inner to this frame (corrupt stack?) (kgdb) what am I doing wrong and what do I have to do to get the specific assert that fails? cheers --m (sorry adriy hI only hit reply the first time not reply-all) On Tue, Jul 26, 2011 at 4:05 AM, Andriy Gapon a...@freebsd.org wrote: on 26/07/2011 04:20 maestro something said the following: Hi, when using the ustack action on the latest FreeBSD8.2 i386 the kernel panics. Here is the information I could gather: let me know if I can provide more information. (i.e., i have the full
