Re: KGDB stack traces in the kernel.
On Monday, April 04, 2011 9:04:23 pm Justin Hibbits wrote: On Apr 4, 2011, at 6:57 PM, Julian Elischer wrote: is there anyone here with enough gdb/kgdb source experience to know what we would need to put on the stack at fork_exit() to make it stop when it gets there? not only is it annoying but it slows down debugging because kgdb and the ddd front end ask for stacks a LOT. sometimes it actually just hangs as the stack goes into a loop and never ends. I had a quick look but didn't spot how gdb decides it has reached the end of a stack. Julian From my experience, it checks for a NULL stack chain pointer. Once that reaches NULL, it's the end of the stack. No, I removed that because it broke debugging panics due to NULL function pointers. -- John Baldwin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: KGDB stack traces in the kernel.
On Tuesday, April 05, 2011 4:35:44 pm Navdeep Parhar wrote:
On Tue, Apr 5, 2011 at 1:33 PM, Julian Elischer jul...@freebsd.org wrote:
On 4/4/11 6:04 PM, Justin Hibbits wrote:
On Apr 4, 2011, at 6:57 PM, Julian Elischer wrote:
is there anyone here with enough gdb/kgdb source experience to know what
we would need to put on the stack at fork_exit() to make it stop when it
gets there?
not only is it annoying but it slows down debugging because kgdb and the
ddd
front end ask for stacks a LOT. sometimes it actually just hangs as the
stack
goes into a loop and never ends.
I had a quick look but didn't spot how gdb decides it has reached the
end
of a stack.
Julian
From my experience, it checks for a NULL stack chain pointer. Once that
reaches NULL, it's the end of the stack.
- Justin
I'll try adding NULL when we build the intial stack up.
:-)
What does ddb do? It always seems to get this stuff correct.
ddb knows to stop when it gets to a non-kernel address, and it uses string
compares on function names to identify trap frames. For example in
sys/amd64/amd64/db_trace.c:
if (strcmp(name, calltrap) == 0 ||
strcmp(name, fork_trampoline) == 0 ||
strcmp(name, nmi_calltrap) == 0 ||
strcmp(name, Xdblfault) == 0)
frame_type = TRAP;
Hah, kgdb just needs to be updated (this is from trgt_amd64.c):
const struct frame_unwind *
kgdb_trgt_trapframe_sniffer(struct frame_info *next_frame)
{
char *pname;
CORE_ADDR pc;
pc = frame_pc_unwind(next_frame);
pname = NULL;
find_pc_partial_function(pc, pname, NULL, NULL);
if (pname == NULL)
return (NULL);
if (strcmp(pname, calltrap) == 0 ||
strcmp(pname, nmi_calltrap) == 0 ||
(pname[0] == 'X' pname[1] != '_'))
return (kgdb_trgt_trapframe_unwind);
/* printf(%s: %lx =%s\n, __func__, pc, pname); */
return (NULL);
}
Can probably just add 'fork_trampoline' to that conditional. I think i386
needs a similar fix in kgdb. Not sure about other architectures:
Index: trgt_amd64.c
===
--- trgt_amd64.c(revision 220190)
+++ trgt_amd64.c(working copy)
@@ -184,6 +184,7 @@
if (pname == NULL)
return (NULL);
if (strcmp(pname, calltrap) == 0 ||
+ strcmp(pname, fork_trampoline) == 0 ||
strcmp(pname, nmi_calltrap) == 0 ||
(pname[0] == 'X' pname[1] != '_'))
return (kgdb_trgt_trapframe_unwind);
Index: trgt_i386.c
===
--- trgt_i386.c (revision 220190)
+++ trgt_i386.c (working copy)
@@ -374,6 +374,7 @@
if (strcmp(pname, dblfault_handler) == 0)
return (kgdb_trgt_dblfault_unwind);
if (strcmp(pname, calltrap) == 0 ||
+ strcmp(pname, fork_trampoline) == 0 ||
(pname[0] == 'X' pname[1] != '_'))
return (kgdb_trgt_trapframe_unwind);
/* printf(%s: %llx =%s\n, __func__, pc, pname); */
--
John Baldwin
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: KGDB stack traces in the kernel.
On 4/6/11 5:45 AM, John Baldwin wrote:
On Tuesday, April 05, 2011 4:35:44 pm Navdeep Parhar wrote:
On Tue, Apr 5, 2011 at 1:33 PM, Julian Elischerjul...@freebsd.org wrote:
On 4/4/11 6:04 PM, Justin Hibbits wrote:
What does ddb do? It always seems to get this stuff correct.
ddb knows to stop when it gets to a non-kernel address, and it uses string
compares on function names to identify trap frames. For example in
sys/amd64/amd64/db_trace.c:
if (strcmp(name, calltrap) == 0 ||
strcmp(name, fork_trampoline) == 0 ||
strcmp(name, nmi_calltrap) == 0 ||
strcmp(name, Xdblfault) == 0)
frame_type = TRAP;
Hah, kgdb just needs to be updated (this is from trgt_amd64.c):
const struct frame_unwind *
kgdb_trgt_trapframe_sniffer(struct frame_info *next_frame)
{
char *pname;
CORE_ADDR pc;
pc = frame_pc_unwind(next_frame);
pname = NULL;
find_pc_partial_function(pc,pname, NULL, NULL);
if (pname == NULL)
return (NULL);
if (strcmp(pname, calltrap) == 0 ||
strcmp(pname, nmi_calltrap) == 0 ||
(pname[0] == 'X' pname[1] != '_'))
return (kgdb_trgt_trapframe_unwind);
/* printf(%s: %lx =%s\n, __func__, pc, pname); */
return (NULL);
}
I'll give that a try
Can probably just add 'fork_trampoline' to that conditional. I think i386
needs a similar fix in kgdb. Not sure about other architectures:
Index: trgt_amd64.c
===
--- trgt_amd64.c(revision 220190)
+++ trgt_amd64.c(working copy)
@@ -184,6 +184,7 @@
if (pname == NULL)
return (NULL);
if (strcmp(pname, calltrap) == 0 ||
+ strcmp(pname, fork_trampoline) == 0 ||
strcmp(pname, nmi_calltrap) == 0 ||
(pname[0] == 'X' pname[1] != '_'))
return (kgdb_trgt_trapframe_unwind);
Index: trgt_i386.c
===
--- trgt_i386.c (revision 220190)
+++ trgt_i386.c (working copy)
@@ -374,6 +374,7 @@
if (strcmp(pname, dblfault_handler) == 0)
return (kgdb_trgt_dblfault_unwind);
if (strcmp(pname, calltrap) == 0 ||
+ strcmp(pname, fork_trampoline) == 0 ||
(pname[0] == 'X' pname[1] != '_'))
return (kgdb_trgt_trapframe_unwind);
/* printf(%s: %llx =%s\n, __func__, pc, pname); */
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: KGDB stack traces in the kernel.
On 4/6/11 10:17 AM, Julian Elischer wrote:
On 4/6/11 5:45 AM, John Baldwin wrote:
On Tuesday, April 05, 2011 4:35:44 pm Navdeep Parhar wrote:
On Tue, Apr 5, 2011 at 1:33 PM, Julian
Elischerjul...@freebsd.org wrote:
On 4/4/11 6:04 PM, Justin Hibbits wrote:
What does ddb do? It always seems to get this stuff correct.
ddb knows to stop when it gets to a non-kernel address, and it uses
string
compares on function names to identify trap frames. For example in
sys/amd64/amd64/db_trace.c:
if (strcmp(name, calltrap) == 0 ||
strcmp(name, fork_trampoline) == 0 ||
strcmp(name, nmi_calltrap) == 0 ||
strcmp(name, Xdblfault) == 0)
frame_type = TRAP;
Hah, kgdb just needs to be updated (this is from trgt_amd64.c):
const struct frame_unwind *
kgdb_trgt_trapframe_sniffer(struct frame_info *next_frame)
{
char *pname;
CORE_ADDR pc;
pc = frame_pc_unwind(next_frame);
pname = NULL;
find_pc_partial_function(pc,pname, NULL, NULL);
if (pname == NULL)
return (NULL);
if (strcmp(pname, calltrap) == 0 ||
strcmp(pname, nmi_calltrap) == 0 ||
(pname[0] == 'X' pname[1] != '_'))
return (kgdb_trgt_trapframe_unwind);
/* printf(%s: %lx =%s\n, __func__, pc, pname); */
return (NULL);
}
I'll give that a try
Can probably just add 'fork_trampoline' to that conditional. I
think i386
needs a similar fix in kgdb. Not sure about other architectures:
Index: trgt_amd64.c
===
--- trgt_amd64.c(revision 220190)
+++ trgt_amd64.c(working copy)
@@ -184,6 +184,7 @@
if (pname == NULL)
return (NULL);
if (strcmp(pname, calltrap) == 0 ||
+strcmp(pname, fork_trampoline) == 0 ||
BTW this is in 8.x
I don't have 9 on a machine I can test this with.
While ddb finds fork_trampoline, (k)gdb does not.
so I had to add fork_exit a well, but then it gets an error as well.
firstly without fork_exit()
#0 0x8144c529 in () from blah.ko
#1 0x814dfea5 in () from blah.ko
#2 0xff0063a1f600 in ?? ()
#3 0x814dfe30 in () from blah.ko
#4 0xff0063a265f0 in ?? ()
#5 0xff8244290c30 in ?? ()
#6 0x805c4b8e in fork_exit (callout=0x814dfe30
XXX, arg=0x89e475f7, frame=0xbad089e44589) at
../../../kern/kern_fork.c:859
#7 0xf445c7a07d894868 in ?? ()
#8 0x095d77e8 in ?? ()
#9 0x00095c40e8c78900 in ?? ()
#10 0x01bfd8458948 in ?? ()
[... etc forever]
if I add for_exit, it does work but:
#6 0xff82441c8c30 in ?? ()
#7 0x805c4b8e in fork_exit (callout=0x814dfe30
XXX, arg=0x89e475f7, frame=0xbad089e44589) at
../../../kern/kern_fork.c:859
Previous frame identical to this frame (corrupt stack?)
that last error message makes ddd throw up its hands and get confused.
it seems to occur all the time however. for example when you use
sysctl debug.kdb.enter to enter the debugger,
the same thing occurs: this easy to see.
#0 kdb_enter (why=0x80a3494c sysctl, msg=0xa Address 0xa
out of bounds) at ../../../kern/subr_kdb.c:352
#1 0x80626bc9 in kdb_sysctl_enter (oidp=0x80c6c360,
arg1=Variable arg1 is not available.
) at ../../../kern/subr_kdb.c:174
#2 0x805fddc3 in sysctl_root (oidp=Variable oidp is not
available.
) at ../../../kern/kern_sysctl.c:1420
#3 0x805fe00e in userland_sysctl (td=0xff0009a7c000,
name=0xff824433fa20, namelen=3, old=0x0, oldlenp=Variable
oldlenp is not available.
) at ../../../kern/kern_sysctl.c:1524
#4 0x805fe4ba in __sysctl (td=0xff0009a7c000,
uap=0xff824433fbb0) at ../../../kern/kern_sysctl.c:1450
#5 0x8063429f in syscallenter (td=0xff0009a7c000,
sa=0xff824433fba0) at ../../../kern/subr_trap.c:315
#6 0x808feafc in syscall (frame=0xff824433fc40) at
../../../amd64/amd64/trap.c:888
#7 0x808e6092 in Xfast_syscall () at
../../../amd64/amd64/exception.S:377
#8 0x00080073db0c in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb)
do you have any idea what that is all about?
looking at frames 7,8,6 I see:
Previous frame inner to this frame (corrupt stack?)
(kgdb) up 7
#7 0x808e6092 in Xfast_syscall () at
../../../amd64/amd64/exception.S:377
377 callsyscall
Current language: auto; currently asm
(kgdb) info frame
Stack level 7, frame at 0xff824433fc40:
rip = 0x808e6092 in Xfast_syscall
(../../../amd64/amd64/exception.S:377); saved rip 0x80073db0c
called by frame at 0x7fffe190, caller of frame at 0xff824433fc40
source language asm.
Arglist at 0xff824433fc38, args:
Locals at 0xff824433fc38, Previous frame's sp at 0xff824433fcf0
Saved registers:
rax at
Re: KGDB stack traces in the kernel.
On 4/4/11 6:04 PM, Justin Hibbits wrote: On Apr 4, 2011, at 6:57 PM, Julian Elischer wrote: is there anyone here with enough gdb/kgdb source experience to know what we would need to put on the stack at fork_exit() to make it stop when it gets there? not only is it annoying but it slows down debugging because kgdb and the ddd front end ask for stacks a LOT. sometimes it actually just hangs as the stack goes into a loop and never ends. I had a quick look but didn't spot how gdb decides it has reached the end of a stack. Julian From my experience, it checks for a NULL stack chain pointer. Once that reaches NULL, it's the end of the stack. - Justin I'll try adding NULL when we build the intial stack up. :-) ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: KGDB stack traces in the kernel.
On Tue, Apr 5, 2011 at 1:33 PM, Julian Elischer jul...@freebsd.org wrote: On 4/4/11 6:04 PM, Justin Hibbits wrote: On Apr 4, 2011, at 6:57 PM, Julian Elischer wrote: is there anyone here with enough gdb/kgdb source experience to know what we would need to put on the stack at fork_exit() to make it stop when it gets there? not only is it annoying but it slows down debugging because kgdb and the ddd front end ask for stacks a LOT. sometimes it actually just hangs as the stack goes into a loop and never ends. I had a quick look but didn't spot how gdb decides it has reached the end of a stack. Julian From my experience, it checks for a NULL stack chain pointer. Once that reaches NULL, it's the end of the stack. - Justin I'll try adding NULL when we build the intial stack up. :-) What does ddb do? It always seems to get this stuff correct. Navdeep ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: KGDB stack traces in the kernel.
On 4/5/11 1:35 PM, Navdeep Parhar wrote: On Tue, Apr 5, 2011 at 1:33 PM, Julian Elischerjul...@freebsd.org wrote: On 4/4/11 6:04 PM, Justin Hibbits wrote: On Apr 4, 2011, at 6:57 PM, Julian Elischer wrote: is there anyone here with enough gdb/kgdb source experience to know what we would need to put on the stack at fork_exit() to make it stop when it gets there? not only is it annoying but it slows down debugging because kgdb and the ddd front end ask for stacks a LOT. sometimes it actually just hangs as the stack goes into a loop and never ends. I had a quick look but didn't spot how gdb decides it has reached the end of a stack. Julian From my experience, it checks for a NULL stack chain pointer. Once that reaches NULL, it's the end of the stack. - Justin I'll try adding NULL when we build the intial stack up. :-) What does ddb do? It always seems to get this stuff correct. Navdeep it has all sorts of special code that knows the intricacies of freebsd's stack. (k)gdb only knows more generic stuff. hmmm set 0 into the saved ebp bit that didn't help.. I'll play a bit more ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
KGDB stack traces in the kernel.
currently in kernel stack traces I see: [Switching to Thread 100246] 0x814617f3 in fio_local_to_global_packet () from x/mumble.ko (kgdb) bt #0 0x814617f3 in () from x/mumble.ko #1 0x8146171f in () from x/mumble.ko [...] #14 0x805c4b8e in fork_exit (callout=0x814dfe30 mumble_kthread_wrapper, arg=0x7d8b48000421dbe8, frame=0x814fc340c6c74800) at ../../../kern/kern_fork.c:859 #15 0xf045c748e87d8948 in ?? () #16 0xb5058b48 in ?? () #17 0x000130054800074e in ?? () #18 0x950fc0850c408b00 in ?? () and then the stack trace goes on forever with garbage. #19 0x74c08548c0b60fc0 in ?? () #20 0x814fdb70c6c74816 in ?? () #21 0xb80007bf in ?? () #22 0x48000425b9e8 in ?? () #23 0x1210c78148e87d8b in ?? () #24 0x480bd1ba in ?? () #25 0x3de8814fc340c6c7 in ?? () #26 0x558b4828eb000421 in ?? () #27 0xc68148e8758b48f0 in ?? () #28 0xe87d8b481210 in ?? () #29 0xe81178c78148 in ?? () #30 0x83fc458900041bdc in ?? () #31 0x7d8b480d7400fc7d in ?? () #32 0xc085fdfae8e8 in ?? () #33 0x8148e87d8b48cb74 in ?? () #34 0x0bdcba1210c7 in ?? () #35 0x4fc340c6c748 in ?? () #36 0x8b48000420aae881 in ?? () #37 0x07600548e845 in ?? () #38 0x0019bfc68948 in ?? () etc is there anyone here with enough gdb/kgdb source experience to know what we would need to put on the stack at fork_exit() to make it stop when it gets there? not only is it annoying but it slows down debugging because kgdb and the ddd front end ask for stacks a LOT. sometimes it actually just hangs as the stack goes into a loop and never ends. I had a quick look but didn't spot how gdb decides it has reached the end of a stack. Julian ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: KGDB stack traces in the kernel.
On Apr 4, 2011, at 6:57 PM, Julian Elischer wrote: is there anyone here with enough gdb/kgdb source experience to know what we would need to put on the stack at fork_exit() to make it stop when it gets there? not only is it annoying but it slows down debugging because kgdb and the ddd front end ask for stacks a LOT. sometimes it actually just hangs as the stack goes into a loop and never ends. I had a quick look but didn't spot how gdb decides it has reached the end of a stack. Julian From my experience, it checks for a NULL stack chain pointer. Once that reaches NULL, it's the end of the stack. - Justin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
