Re: panic: absolutely cannot call smp_ipi_shootdown with interrupts already disabled
At 4:47 PM -0700 2003/09/17, Doug White wrote: This came up at the developer summit. We do need to upgrade/make significant changes to gdb for it to understand threaded debugging. The panics might be interesting as it might be tickling other issues, but before we can really debug threaded apps we need a new gdb. Do we need a rewritten gdb, or can we have both the current gdb and a new tgdb for debugging the threaded stuff? The latter approach seems to be something that would be a lot easier to integrate without risk of breaking anything currently existing, and tgdb could even be done as a port until such time as it's fully ready to take over from the built-in gdb in the system. -- Brad Knowles, [EMAIL PROTECTED] They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety. -Benjamin Franklin, Historical Review of Pennsylvania. GCS/IT d+(-) s:+(++): a C++(+++)$ UMBSHI$ P+++ L+ !E-(---) W+++(--) N+ !w--- O- M++ V PS++(+++) PE- Y+(++) PGP+++ t+(+++) 5++(+++) X++(+++) R+(+++) tv+(+++) b+() DI+() D+(++) G+() e++ h--- r---(+++)* z(+++) ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: panic: absolutely cannot call smp_ipi_shootdown with interrupts already disabled
On Thu, 18 Sep 2003, Brad Knowles wrote: At 4:47 PM -0700 2003/09/17, Doug White wrote: This came up at the developer summit. We do need to upgrade/make significant changes to gdb for it to understand threaded debugging. The panics might be interesting as it might be tickling other issues, but before we can really debug threaded apps we need a new gdb. The panics had nothing to do with KSE, gdb, or threading and were recently fixed. Do we need a rewritten gdb, or can we have both the current gdb and a new tgdb for debugging the threaded stuff? It is relatively easy to modify the existing threading support in gdb so that we can at least view the current threads. Other kernel support would be needed to attach to threads in other KSEs and retrieve register sets for threads blocked in the kernel. Libthr has similar problems. The latter approach seems to be something that would be a lot easier to integrate without risk of breaking anything currently existing, and tgdb could even be done as a port until such time as it's fully ready to take over from the built-in gdb in the system. FreeBSD's threading support is not presently in stock GDB anyways; it's in gnu/usr.bin/binutils/gdb/freebsd-uthread.c, not in contrib/gdb/... I don't see the need for a separate port. The two problems are: o adding kernel support to attach to threads in different KSEs and fetching register sets from threads blocked in the kernel o making gdb understand not only libc_r, but also libthr and libpthread, and how to determine which library it is debugging. -- Dan Eischen ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: panic: absolutely cannot call smp_ipi_shootdown with interrupts already disabled
On Tue, 16 Sep 2003, Morten Rodal wrote: So this brings up the question, is there a known problem with debugging multi-threaded applications (which use libkse) that can cause a panic? I will bring home my laptop, and will be able to use a serial debugger if that would help anyone willing to trace this down. Probably :) This came up at the developer summit. We do need to upgrade/make significant changes to gdb for it to understand threaded debugging. The panics might be interesting as it might be tickling other issues, but before we can really debug threaded apps we need a new gdb. -- Doug White| FreeBSD: The Power to Serve [EMAIL PROTECTED] | www.FreeBSD.org ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: panic: absolutely cannot call smp_ipi_shootdown with interrupts already disabled
On Fri, Sep 12, 2003 at 08:54:59AM +0200, Morten Rodal wrote: A little bit of history first. I am having great trouble in running any of the Mozilla web browsers under -CURRENT with libkse. (If you are really interested see the thread on threads@) When I ran Mozilla Firebird with the --debug (which lets you run Mozilla Firebird from within gdb) the machine paniced. The backtrace is rather long and I am not sure if the subject of this email is the real panic either. This computer is/was (I am currently rebuilding the kernel to todays sources) running: FreeBSD slurp.rodal.no 5.1-CURRENT FreeBSD 5.1-CURRENT #1: Fri Aug 22 18:06:03 CEST 2003 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/slurp i386 More details are available upon request. Since the last panic I upgraded to FreeBSD slurp.rodal.no 5.1-CURRENT FreeBSD 5.1-CURRENT #2: Fri Sep 12 08:59:58 CEST 2003 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/slurp i386 (both world and kernel). I am still able to reproduce this, and it is fairly simple. On a smp machine (haven't tried my laptop, and I got more important stuff there that I dont want to lose in a panic) do the following: 1) Install the mozilla-firebird port 2) Edit libmap.conf so that it uses libkse 3) Start it by running firebird --debug, this will present you with a gdb prompt. Type run here and watch your computer panic. (It takes a little while) So this brings up the question, is there a known problem with debugging multi-threaded applications (which use libkse) that can cause a panic? I will bring home my laptop, and will be able to use a serial debugger if that would help anyone willing to trace this down. -- Morten Rodal Script started on Tue Sep 16 08:25:54 2003 slurp# gdb -k kernel.13 vmcore.13 GNU gdb 5.2.1 (FreeBSD) Copyright 2002 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type show copying to see the conditions. There is absolutely no warranty for GDB. Type show warranty for details. This GDB was configured as i386-undermydesk-freebsd... panic: absolutely cannot call smp_ipi_shootdown with interrupts already disabled panic messages: --- panic: absolutely cannot call smp_ipi_shootdown with interrupts already disabled cpuid = 0; lapic.id = 0100 Stack backtrace: boot() called on cpu#0 syncing disks, buffers remaining... panic: absolutely cannot call smp_ipi_shootdown with interrupts already disabled cpuid = 0; lapic.id = 0100 boot() called on cpu#0 Uptime: 3d19h39m32s Dumping 447 MB 16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 256 272 288 304 320 336 352 368 384 400 416 432 --- Reading symbols from /usr/obj/usr/src/sys/slurp/modules/usr/src/sys/modules/linux/linux.ko.debug...done. Loaded symbols for /usr/obj/usr/src/sys/slurp/modules/usr/src/sys/modules/linux/linux.ko.debug Reading symbols from /boot/kernel/snd_sb16.ko...done. Loaded symbols for /boot/kernel/snd_sb16.ko Reading symbols from /boot/kernel/snd_sbc.ko...done. Loaded symbols for /boot/kernel/snd_sbc.ko Reading symbols from /boot/kernel/snd_pcm.ko...done. Loaded symbols for /boot/kernel/snd_pcm.ko Reading symbols from /usr/obj/usr/src/sys/slurp/modules/usr/src/sys/modules/acpi/acpi.ko.debug...done. Loaded symbols for /usr/obj/usr/src/sys/slurp/modules/usr/src/sys/modules/acpi/acpi.ko.debug Reading symbols from /usr/obj/usr/src/sys/slurp/modules/usr/src/sys/modules/if_gif/if_gif.ko.debug...done. Loaded symbols for /usr/obj/usr/src/sys/slurp/modules/usr/src/sys/modules/if_gif/if_gif.ko.debug Reading symbols from /boot/kernel/nvidia.ko...done. Loaded symbols for /boot/kernel/nvidia.ko Reading symbols from /usr/obj/usr/src/sys/slurp/modules/usr/src/sys/modules/cd9660/cd9660.ko.debug...done. Loaded symbols for /usr/obj/usr/src/sys/slurp/modules/usr/src/sys/modules/cd9660/cd9660.ko.debug Reading symbols from /usr/obj/usr/src/sys/slurp/modules/usr/src/sys/modules/nfsserver/nfsserver.ko.debug...done. Loaded symbols for /usr/obj/usr/src/sys/slurp/modules/usr/src/sys/modules/nfsserver/nfsserver.ko.debug #0 doadump () at /usr/src/sys/kern/kern_shutdown.c:240 240 dumping++; (kgdb) bt #0 doadump () at /usr/src/sys/kern/kern_shutdown.c:240 #1 0xc01de9b6 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:372 #2 0xc01dee08 in panic () at /usr/src/sys/kern/kern_shutdown.c:550 #3 0xc0320467 in smp_tlb_shootdown (vector=0, addr1=0, addr2=0) at /usr/src/sys/i386/i386/mp_machdep.c:2396 #4 0xc03207a9 in smp_invlpg_range (addr1=0, addr2=0) at /usr/src/sys/i386/i386/mp_machdep.c:2527 #5 0xc0322b38 in pmap_invalidate_range (pmap=0xc03eda40, sva=3440447488, eva=3440463872) at /usr/src/sys/i386/i386/pmap.c:719 #6 0xc0323011 in pmap_qremove (sva=3440447488, count=0) at /usr/src/sys/i386/i386/pmap.c:985 #7 0xc022ba4a in vfs_vmio_release (bp=0xcca1ec60) at /usr/src/sys/kern/vfs_bio.c:1588 #8
panic: absolutely cannot call smp_ipi_shootdown with interrupts already disabled
A little bit of history first. I am having great trouble in running any of the Mozilla web browsers under -CURRENT with libkse. (If you are really interested see the thread on threads@) When I ran Mozilla Firebird with the --debug (which lets you run Mozilla Firebird from within gdb) the machine paniced. The backtrace is rather long and I am not sure if the subject of this email is the real panic either. This computer is/was (I am currently rebuilding the kernel to todays sources) running: FreeBSD slurp.rodal.no 5.1-CURRENT FreeBSD 5.1-CURRENT #1: Fri Aug 22 18:06:03 CEST 2003 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/slurp i386 More details are available upon request. -- Morten Rodal Script started on Fri Sep 12 08:49:49 2003 slurp# gdb -k kernel.12 vmcore.12 GNU gdb 5.2.1 (FreeBSD) Copyright 2002 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type show copying to see the conditions. There is absolutely no warranty for GDB. Type show warranty for details. This GDB was configured as i386-undermydesk-freebsd... panic: absolutely cannot call smp_ipi_shootdown with interrupts already disabled panic messages: --- panic: absolutely cannot call smp_ipi_shootdown with interrupts already disabled cpuid = 1; lapic.id = Stack backtrace: boot() called on cpu#1 syncing disks, buffers remaining... panic: absolutely cannot call smp_ipi_shootdown with interrupts already disabled cpuid = 1; lapic.id = boot() called on cpu#1 Uptime: 3d0h0m11s Dumping 447 MB [CTRL-C to abort] [CTRL-C to abort] [CTRL-C to abort] [CTRL-C to abort] [CTRL-C to abort] 16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 256 272 288 304 320 336 352 368 384 400 416 432 --- Reading symbols from /boot/kernel/linux.ko...done. Loaded symbols for /boot/kernel/linux.ko Reading symbols from /boot/kernel/snd_sb16.ko...done. Loaded symbols for /boot/kernel/snd_sb16.ko Reading symbols from /boot/kernel/snd_sbc.ko...done. Loaded symbols for /boot/kernel/snd_sbc.ko Reading symbols from /boot/kernel/snd_pcm.ko...done. Loaded symbols for /boot/kernel/snd_pcm.ko Reading symbols from /boot/kernel/nvidia.ko...done. Loaded symbols for /boot/kernel/nvidia.ko Reading symbols from /boot/kernel/acpi.ko...done. Loaded symbols for /boot/kernel/acpi.ko Reading symbols from /boot/kernel/if_gif.ko...done. Loaded symbols for /boot/kernel/if_gif.ko Reading symbols from /boot/kernel/nfsserver.ko...done. Loaded symbols for /boot/kernel/nfsserver.ko Reading symbols from /boot/kernel/cd9660.ko...done. Loaded symbols for /boot/kernel/cd9660.ko #0 doadump () at /usr/src/sys/kern/kern_shutdown.c:240 240 dumping++; (kgdb) bt #0 doadump () at /usr/src/sys/kern/kern_shutdown.c:240 #1 0xc01e40f6 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:372 #2 0xc01e4548 in panic () at /usr/src/sys/kern/kern_shutdown.c:550 #3 0xc0322347 in smp_tlb_shootdown (vector=0, addr1=0, addr2=0) at /usr/src/sys/i386/i386/mp_machdep.c:2397 #4 0xc0322689 in smp_invlpg_range (addr1=0, addr2=0) at /usr/src/sys/i386/i386/mp_machdep.c:2529 #5 0xc0324a18 in pmap_invalidate_range (pmap=0xc03e6640, sva=3489652736, eva=3489669120) at /usr/src/sys/i386/i386/pmap.c:719 #6 0xc0324d71 in pmap_qremove (sva=3489652736, count=0) at /usr/src/sys/i386/i386/pmap.c:961 #7 0xc022ffaa in vfs_vmio_release (bp=0xccab1058) at /usr/src/sys/kern/vfs_bio.c:1587 #8 0xc0230724 in getnewbuf (slpflag=0, slptimeo=0, size=6144, maxsize=16384) at /usr/src/sys/kern/vfs_bio.c:1864 #9 0xc0231fba in geteblk (size=6144) at /usr/src/sys/kern/vfs_bio.c:2627 #10 0xc022dca3 in bwrite (bp=0x4000) at /usr/src/sys/kern/vfs_bio.c:813 #11 0xc022eabc in bawrite (bp=0x0) at /usr/src/sys/kern/vfs_bio.c:1148 #12 0xc0238bc0 in vop_stdfsync (ap=0xdc2bc708) at /usr/src/sys/kern/vfs_default.c:742 #13 0xc01a8ad0 in spec_fsync (ap=0xdc2bc708) at /usr/src/sys/fs/specfs/spec_vnops.c:417 #14 0xc01a7ec8 in spec_vnoperate (ap=0x0) at /usr/src/sys/fs/specfs/spec_vnops.c:122 #15 0xc02c1fe1 in ffs_sync (mp=0xc3a2ec00, waitfor=2, cred=0xc1378e80, td=0xc03adcc0) at vnode_if.h:627 #16 0xc0246aeb in sync (td=0xc03adcc0, uap=0x0) at /usr/src/sys/kern/vfs_syscalls.c:142 ---Type return to continue, or q return to quit--- #17 0xc01e3bff in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:281 #18 0xc01e4548 in panic () at /usr/src/sys/kern/kern_shutdown.c:550 #19 0xc0322347 in smp_tlb_shootdown (vector=0, addr1=0, addr2=0) at /usr/src/sys/i386/i386/mp_machdep.c:2397 #20 0xc032264a in smp_invlpg (addr=0) at /usr/src/sys/i386/i386/mp_machdep.c:2516 #21 0xc0324993 in pmap_invalidate_page (pmap=0xc03e6640, va=3313360896) at /usr/src/sys/i386/i386/pmap.c:691 #22 0xc03267c9 in pmap_enter (pmap=0xc03e6640, va=3225314880, m=0xc0baac48, prot=7 '\a', wired=1) at /usr/src/sys/i386/i386/pmap.c:2037 #23 0xc02dba67
Re: panic: absolutely cannot call smp_ipi_shootdown with interrupts already disabled
Peter Wemm writes: Eww! Care to confirm that the following works? I was going to just commit it since it is pretty obvious, but a brief sanity check would probably be an idea. (beware, xterm cut/paste whitespace damage). ddb runs with interrupts disabled and the other cpus halted. We could not get the 'ack' from the IPI. The panic was real - you'd have locked up hard if it had not triggered. db_write_bytes temporarily changes a local cpu mapping and changes it back while everything else is frozen. We do not need to do a coherent invalidate. That seems to work. Thanks! Drew To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
