Re: DEADLKRES crash
On 8/19/2014 8:53 AM, Larry Rosenman wrote: > On 2014-08-19 08:42, Eric van Gyzen wrote: >> On 08/18/2014 16:45, Ryan Stone wrote: >>> The first thing that I'd like to see is (in kgdb): >>> >>> set $td=(struct thread)0xf8002abeb000 >>> tid $td->td_tid >>> bt >>> >>> That will show us the backtrace of the thread that was blocked for so >>> long. >> >> Make that: >> >> set $td=(struct thread *)0xf8002abeb000 >> tid $td->td_tid >> bt >> >> >> Eric > #0 doadump (textdump=1) at pcpu.h:219 > 219pcpu.h: No such file or directory. > in pcpu.h > (kgdb) set $td=(struct thread *)0xf8002abeb000 > Current language: auto; currently minimal > (kgdb) tid $td->td_tid > [Switching to thread 469 (Thread 100681)]#0 sched_switch ( > td=0xf8002abeb000, newtd=, > flags=) at /usr/src/sys/kern/sched_ule.c:1931 > 1931cpuid = PCPU_GET(cpuid); > (kgdb) bt > #0 sched_switch (td=0xf8002abeb000, newtd=, > flags=) at /usr/src/sys/kern/sched_ule.c:1931 > #1 0x80a107d9 in mi_switch (flags=260, newtd=0x0) > at /usr/src/sys/kern/kern_synch.c:493 > #2 0x80a4c442 in sleepq_switch (wchan=, > pri=) at /usr/src/sys/kern/subr_sleepqueue.c:552 > #3 0x80a4c2a3 in sleepq_wait (wchan=0xf80070a4dd50, pri=96) > at /usr/src/sys/kern/subr_sleepqueue.c:631 > #4 0x809eb1fa in sleeplk (lk=, > flags=, ilk=, > wmesg=, pri=, > timo=) at /usr/src/sys/kern/kern_lock.c:225 > #5 0x809eaa06 in __lockmgr_args (lk=0xf80070a4dd50, > flags=, ilk=0xf80070a4dd80, > wmesg=, pri=, > timo=) at /usr/src/sys/kern/kern_lock.c:931 > #6 0x8092e092 in nfs_lock1 (ap=) at > lockmgr.h:97 > #7 0x80f2d57c in VOP_LOCK1_APV (vop=, > a=) at vnode_if.c:2082 > #8 0x80abd22a in _vn_lock (vp=0xf80070a4dce8, > flags=, > file=0x8110db88 "/usr/src/sys/kern/vfs_subr.c", line=2137) > at vnode_if.h:859 > #9 0x80aad4e7 in vget (vp=0xf80070a4dce8, flags=524544, > ---Type to continue, or q to quit--- > td=0xf8002abeb000) at /usr/src/sys/kern/vfs_subr.c:2137 > #10 0x80aa1491 in vfs_hash_get (mp=0xf8002aa1e990, > hash=1741450670, flags=, td=0xf8002abeb000, > vpp=0xfe100c75c670, fn=0x80935820 ) > at /usr/src/sys/kern/vfs_hash.c:88 > #11 0x809314bd in ncl_nget (mntp=0xf8002aa1e990, > fhp=0xf80070ccf4a4 "\001", fhsize=12, npp=0xfe100c75c6e0, > lkflags=) > at /usr/src/sys/fs/nfsclient/nfs_clnode.c:114 > #12 0x809340fd in nfs_statfs (mp=0xf8002aa1e990, > sbp=0xf8002aa1ea48) at /usr/src/sys/fs/nfsclient/nfs_clvfsops.c:288 > #13 0x80aa7ade in __vfs_statfs (mp=0x0, sbp=0xf8002aa1ea48) > at /usr/src/sys/kern/vfs_mount.c:1706 > #14 0x80ab4f5e in kern_getfsstat (td=0xf8002abeb000, > buf=, bufsize=, > bufseg=UIO_USERSPACE, flags=) > at /usr/src/sys/kern/vfs_syscalls.c:511 > #15 0x80e1625a in amd64_syscall (td=0xf8002abeb000, traced=0) > at subr_syscall.c:133 > #16 0x80df760b in Xfast_syscall () > at /usr/src/sys/amd64/amd64/exception.S:390 > #17 0x0008010fc83a in ?? () > Previous frame inner to this frame (corrupt stack?) > (kgdb) Looks like the one I hit recently as well: http://lists.freebsd.org/pipermail/freebsd-fs/2014-July/019843.html This lock should probably be excluded from the DEADLKRES. I have not had time to follow-up on it, but it's a trivial thing to add it to the list. -- Regards, Bryan Drewery signature.asc Description: OpenPGP digital signature
Re: DEADLKRES crash
On 2014-08-19 08:42, Eric van Gyzen wrote: On 08/18/2014 16:45, Ryan Stone wrote: The first thing that I'd like to see is (in kgdb): set $td=(struct thread)0xf8002abeb000 tid $td->td_tid bt That will show us the backtrace of the thread that was blocked for so long. Make that: set $td=(struct thread *)0xf8002abeb000 tid $td->td_tid bt Eric #0 doadump (textdump=1) at pcpu.h:219 219 pcpu.h: No such file or directory. in pcpu.h (kgdb) set $td=(struct thread *)0xf8002abeb000 Current language: auto; currently minimal (kgdb) tid $td->td_tid [Switching to thread 469 (Thread 100681)]#0 sched_switch ( td=0xf8002abeb000, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1931 1931cpuid = PCPU_GET(cpuid); (kgdb) bt #0 sched_switch (td=0xf8002abeb000, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1931 #1 0x80a107d9 in mi_switch (flags=260, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:493 #2 0x80a4c442 in sleepq_switch (wchan=, pri=) at /usr/src/sys/kern/subr_sleepqueue.c:552 #3 0x80a4c2a3 in sleepq_wait (wchan=0xf80070a4dd50, pri=96) at /usr/src/sys/kern/subr_sleepqueue.c:631 #4 0x809eb1fa in sleeplk (lk=, flags=, ilk=, wmesg=, pri=, timo=) at /usr/src/sys/kern/kern_lock.c:225 #5 0x809eaa06 in __lockmgr_args (lk=0xf80070a4dd50, flags=, ilk=0xf80070a4dd80, wmesg=, pri=, timo=) at /usr/src/sys/kern/kern_lock.c:931 #6 0x8092e092 in nfs_lock1 (ap=) at lockmgr.h:97 #7 0x80f2d57c in VOP_LOCK1_APV (vop=, a=) at vnode_if.c:2082 #8 0x80abd22a in _vn_lock (vp=0xf80070a4dce8, flags=, file=0x8110db88 "/usr/src/sys/kern/vfs_subr.c", line=2137) at vnode_if.h:859 #9 0x80aad4e7 in vget (vp=0xf80070a4dce8, flags=524544, ---Type to continue, or q to quit--- td=0xf8002abeb000) at /usr/src/sys/kern/vfs_subr.c:2137 #10 0x80aa1491 in vfs_hash_get (mp=0xf8002aa1e990, hash=1741450670, flags=, td=0xf8002abeb000, vpp=0xfe100c75c670, fn=0x80935820 ) at /usr/src/sys/kern/vfs_hash.c:88 #11 0x809314bd in ncl_nget (mntp=0xf8002aa1e990, fhp=0xf80070ccf4a4 "\001", fhsize=12, npp=0xfe100c75c6e0, lkflags=) at /usr/src/sys/fs/nfsclient/nfs_clnode.c:114 #12 0x809340fd in nfs_statfs (mp=0xf8002aa1e990, sbp=0xf8002aa1ea48) at /usr/src/sys/fs/nfsclient/nfs_clvfsops.c:288 #13 0x80aa7ade in __vfs_statfs (mp=0x0, sbp=0xf8002aa1ea48) at /usr/src/sys/kern/vfs_mount.c:1706 #14 0x80ab4f5e in kern_getfsstat (td=0xf8002abeb000, buf=, bufsize=, bufseg=UIO_USERSPACE, flags=) at /usr/src/sys/kern/vfs_syscalls.c:511 #15 0x80e1625a in amd64_syscall (td=0xf8002abeb000, traced=0) at subr_syscall.c:133 #16 0x80df760b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:390 #17 0x0008010fc83a in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: l...@lerctr.org US Mail: 108 Turvey Cove, Hutto, TX 78634-5688 ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: DEADLKRES crash
On 08/18/2014 16:45, Ryan Stone wrote: > The first thing that I'd like to see is (in kgdb): > > set $td=(struct thread)0xf8002abeb000 > tid $td->td_tid > bt > > That will show us the backtrace of the thread that was blocked for so long. Make that: set $td=(struct thread *)0xf8002abeb000 tid $td->td_tid bt Eric ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: DEADLKRES crash
On 2014-08-18 15:45, Ryan Stone wrote: On Mon, Aug 18, 2014 at 11:21 AM, Larry Rosenman wrote: I got the following: borg.lerctr.org dumped core - see /var/crash/vmcore.8 Mon Aug 18 07:30:42 CDT 2014 FreeBSD borg.lerctr.org 11.0-CURRENT FreeBSD 11.0-CURRENT #63 r269784M: Sun Aug 10 12:33:07 CDT 2014 r...@borg.lerctr.org:/usr/obj/usr/src/sys/VT-LER amd64 panic: deadlkres: possible deadlock detected for 0xf8002abeb000, blocked for 1800926 ticks GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: panic: deadlkres: possible deadlock detected for 0xf8002abeb000, blocked for 1800926 ticks cpuid = 3 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfe100bff1a10 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfe100bff1ac0 vpanic() at vpanic+0x126/frame 0xfe100bff1b00 panic() at panic+0x43/frame 0xfe100bff1b60 deadlkres() at deadlkres+0x35c/frame 0xfe100bff1bb0 fork_exit() at fork_exit+0x84/frame 0xfe100bff1bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfe100bff1bf0 --- trap 0, rip = 0, rsp = 0xfe100bff1cb0, rbp = 0 --- Uptime: 7d14h14m38s The first thing that I'd like to see is (in kgdb): set $td=(struct thread)0xf8002abeb000 tid $td->td_tid bt That will show us the backtrace of the thread that was blocked for so long. 0 doadump (textdump=1) at pcpu.h:219 219 pcpu.h: No such file or directory. in pcpu.h (kgdb) set $td=(struct thread)0xf8002abeb000 Invalid cast. (kgdb) set $td=(struct thread*)0xf8002abeb000 Current language: auto; currently minimal (kgdb) tid $td->td_tid [Switching to thread 469 (Thread 100681)]#0 sched_switch ( td=0xf8002abeb000, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1931 1931cpuid = PCPU_GET(cpuid); (kgdb) bt #0 sched_switch (td=0xf8002abeb000, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1931 #1 0x80a107d9 in mi_switch (flags=260, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:493 #2 0x80a4c442 in sleepq_switch (wchan=, pri=) at /usr/src/sys/kern/subr_sleepqueue.c:552 #3 0x80a4c2a3 in sleepq_wait (wchan=0xf80070a4dd50, pri=96) at /usr/src/sys/kern/subr_sleepqueue.c:631 #4 0x809eb1fa in sleeplk (lk=, flags=, ilk=, wmesg=, pri=, timo=) at /usr/src/sys/kern/kern_lock.c:225 #5 0x809eaa06 in __lockmgr_args (lk=0xf80070a4dd50, flags=, ilk=0xf80070a4dd80, wmesg=, pri=, timo=) at /usr/src/sys/kern/kern_lock.c:931 #6 0x8092e092 in nfs_lock1 (ap=) at lockmgr.h:97 #7 0x80f2d57c in VOP_LOCK1_APV (vop=, a=) at vnode_if.c:2082 #8 0x80abd22a in _vn_lock (vp=0xf80070a4dce8, flags=, file=0x8110db88 "/usr/src/sys/kern/vfs_subr.c", line=2137) at vnode_if.h:859 #9 0x80aad4e7 in vget (vp=0xf80070a4dce8, flags=524544, ---Type to continue, or q to quit--- td=0xf8002abeb000) at /usr/src/sys/kern/vfs_subr.c:2137 #10 0x80aa1491 in vfs_hash_get (mp=0xf8002aa1e990, hash=1741450670, flags=, td=0xf8002abeb000, vpp=0xfe100c75c670, fn=0x80935820 ) at /usr/src/sys/kern/vfs_hash.c:88 #11 0x809314bd in ncl_nget (mntp=0xf8002aa1e990, fhp=0xf80070ccf4a4 "\001", fhsize=12, npp=0xfe100c75c6e0, lkflags=) at /usr/src/sys/fs/nfsclient/nfs_clnode.c:114 #12 0x809340fd in nfs_statfs (mp=0xf8002aa1e990, sbp=0xf8002aa1ea48) at /usr/src/sys/fs/nfsclient/nfs_clvfsops.c:288 #13 0x80aa7ade in __vfs_statfs (mp=0x0, sbp=0xf8002aa1ea48) at /usr/src/sys/kern/vfs_mount.c:1706 #14 0x80ab4f5e in kern_getfsstat (td=0xf8002abeb000, buf=, bufsize=, bufseg=UIO_USERSPACE, flags=) at /usr/src/sys/kern/vfs_syscalls.c:511 #15 0x80e1625a in amd64_syscall (td=0xf8002abeb000, traced=0) at subr_syscall.c:133 #16 0x80df760b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:390 #17 0x0008010fc83a in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: l...@lerctr.org US Mail: 108 Turvey Cove, Hutto, TX 78634-5688 ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: DEADLKRES crash
On Mon, Aug 18, 2014 at 11:21 AM, Larry Rosenman wrote: > I got the following: > > borg.lerctr.org dumped core - see /var/crash/vmcore.8 > > Mon Aug 18 07:30:42 CDT 2014 > > FreeBSD borg.lerctr.org 11.0-CURRENT FreeBSD 11.0-CURRENT #63 r269784M: Sun > Aug 10 12:33:07 CDT 2014 r...@borg.lerctr.org:/usr/obj/usr/src/sys/VT-LER > amd64 > > panic: deadlkres: possible deadlock detected for 0xf8002abeb000, blocked > for 1800926 ticks > > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "amd64-marcel-freebsd"... > > Unread portion of the kernel message buffer: > panic: deadlkres: possible deadlock detected for 0xf8002abeb000, blocked > for 1800926 ticks > > cpuid = 3 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfe100bff1a10 > kdb_backtrace() at kdb_backtrace+0x39/frame 0xfe100bff1ac0 > vpanic() at vpanic+0x126/frame 0xfe100bff1b00 > panic() at panic+0x43/frame 0xfe100bff1b60 > deadlkres() at deadlkres+0x35c/frame 0xfe100bff1bb0 > fork_exit() at fork_exit+0x84/frame 0xfe100bff1bf0 > fork_trampoline() at fork_trampoline+0xe/frame 0xfe100bff1bf0 > --- trap 0, rip = 0, rsp = 0xfe100bff1cb0, rbp = 0 --- > Uptime: 7d14h14m38s The first thing that I'd like to see is (in kgdb): set $td=(struct thread)0xf8002abeb000 tid $td->td_tid bt That will show us the backtrace of the thread that was blocked for so long. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: DEADLKRES crash
On 2014-08-18 12:56, Benjamin Kaduk wrote: On Mon, 18 Aug 2014, Larry Rosenman wrote: I got the following: borg.lerctr.org dumped core - see /var/crash/vmcore.8 Mon Aug 18 07:30:42 CDT 2014 FreeBSD borg.lerctr.org 11.0-CURRENT FreeBSD 11.0-CURRENT #63 r269784M: Sun Aug 10 12:33:07 CDT 2014 r...@borg.lerctr.org:/usr/obj/usr/src/sys/VT-LER amd64 panic: deadlkres: possible deadlock detected for 0xf8002abeb000, blocked for 1800926 ticks [...] Current language: auto; currently minimal (kgdb) What info do folks need? Most useful would be the "show alllocks" from DDB. Getting the lock information from kgdb is rather more annoying, though jhb has some gdb scripts which help. (http://people.freebsd.org/~jhb/gdb/) I guess the 'allchains' command from gdb6 is the one in question, but it's been a while since I tried to use these scripts. -Ben ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" after source'ing the above, and typing allchains, it sits and spins, but no output... :( Ideas? I **CAN** give SSH access -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: l...@lerctr.org US Mail: 108 Turvey Cove, Hutto, TX 78634-5688 ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: DEADLKRES crash
On Mon, 18 Aug 2014, Larry Rosenman wrote: > I got the following: > > borg.lerctr.org dumped core - see /var/crash/vmcore.8 > > Mon Aug 18 07:30:42 CDT 2014 > > FreeBSD borg.lerctr.org 11.0-CURRENT FreeBSD 11.0-CURRENT #63 r269784M: Sun > Aug 10 12:33:07 CDT 2014 r...@borg.lerctr.org:/usr/obj/usr/src/sys/VT-LER > amd64 > > panic: deadlkres: possible deadlock detected for 0xf8002abeb000, > blocked for 1800926 ticks [...] > Current language: auto; currently minimal > (kgdb) > > What info do folks need? Most useful would be the "show alllocks" from DDB. Getting the lock information from kgdb is rather more annoying, though jhb has some gdb scripts which help. (http://people.freebsd.org/~jhb/gdb/) I guess the 'allchains' command from gdb6 is the one in question, but it's been a while since I tried to use these scripts. -Ben ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
DEADLKRES crash
I got the following: borg.lerctr.org dumped core - see /var/crash/vmcore.8 Mon Aug 18 07:30:42 CDT 2014 FreeBSD borg.lerctr.org 11.0-CURRENT FreeBSD 11.0-CURRENT #63 r269784M: Sun Aug 10 12:33:07 CDT 2014 r...@borg.lerctr.org:/usr/obj/usr/src/sys/VT-LER amd64 panic: deadlkres: possible deadlock detected for 0xf8002abeb000, blocked for 1800926 ticks GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: panic: deadlkres: possible deadlock detected for 0xf8002abeb000, blocked for 1800926 ticks cpuid = 3 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfe100bff1a10 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfe100bff1ac0 vpanic() at vpanic+0x126/frame 0xfe100bff1b00 panic() at panic+0x43/frame 0xfe100bff1b60 deadlkres() at deadlkres+0x35c/frame 0xfe100bff1bb0 fork_exit() at fork_exit+0x84/frame 0xfe100bff1bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfe100bff1bf0 --- trap 0, rip = 0, rsp = 0xfe100bff1cb0, rbp = 0 --- Uptime: 7d14h14m38s Dumping 9124 out of 64463 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% Reading symbols from /boot/kernel/linux.ko.symbols...done. Loaded symbols for /boot/kernel/linux.ko.symbols Reading symbols from /boot/kernel/if_lagg.ko.symbols...done. Loaded symbols for /boot/kernel/if_lagg.ko.symbols Reading symbols from /boot/kernel/snd_envy24ht.ko.symbols...done. Loaded symbols for /boot/kernel/snd_envy24ht.ko.symbols Reading symbols from /boot/kernel/snd_spicds.ko.symbols...done. Loaded symbols for /boot/kernel/snd_spicds.ko.symbols Reading symbols from /boot/kernel/coretemp.ko.symbols...done. Loaded symbols for /boot/kernel/coretemp.ko.symbols Reading symbols from /boot/kernel/ichsmb.ko.symbols...done. Loaded symbols for /boot/kernel/ichsmb.ko.symbols Reading symbols from /boot/kernel/smbus.ko.symbols...done. Loaded symbols for /boot/kernel/smbus.ko.symbols Reading symbols from /boot/kernel/ichwd.ko.symbols...done. Loaded symbols for /boot/kernel/ichwd.ko.symbols Reading symbols from /boot/kernel/cpuctl.ko.symbols...done. Loaded symbols for /boot/kernel/cpuctl.ko.symbols Reading symbols from /boot/kernel/crypto.ko.symbols...done. Loaded symbols for /boot/kernel/crypto.ko.symbols Reading symbols from /boot/kernel/cryptodev.ko.symbols...done. Loaded symbols for /boot/kernel/cryptodev.ko.symbols Reading symbols from /boot/kernel/dtraceall.ko.symbols...done. Loaded symbols for /boot/kernel/dtraceall.ko.symbols Reading symbols from /boot/kernel/profile.ko.symbols...done. Loaded symbols for /boot/kernel/profile.ko.symbols Reading symbols from /boot/kernel/cyclic.ko.symbols...done. Loaded symbols for /boot/kernel/cyclic.ko.symbols Reading symbols from /boot/kernel/dtrace.ko.symbols...done. Loaded symbols for /boot/kernel/dtrace.ko.symbols Reading symbols from /boot/kernel/systrace_freebsd32.ko.symbols...done. Loaded symbols for /boot/kernel/systrace_freebsd32.ko.symbols Reading symbols from /boot/kernel/systrace.ko.symbols...done. Loaded symbols for /boot/kernel/systrace.ko.symbols Reading symbols from /boot/kernel/sdt.ko.symbols...done. Loaded symbols for /boot/kernel/sdt.ko.symbols Reading symbols from /boot/kernel/lockstat.ko.symbols...done. Loaded symbols for /boot/kernel/lockstat.ko.symbols Reading symbols from /boot/kernel/fasttrap.ko.symbols...done. Loaded symbols for /boot/kernel/fasttrap.ko.symbols Reading symbols from /boot/kernel/fbt.ko.symbols...done. Loaded symbols for /boot/kernel/fbt.ko.symbols Reading symbols from /boot/kernel/dtnfscl.ko.symbols...done. Loaded symbols for /boot/kernel/dtnfscl.ko.symbols Reading symbols from /boot/kernel/dtmalloc.ko.symbols...done. Loaded symbols for /boot/kernel/dtmalloc.ko.symbols Reading symbols from /boot/modules/vboxdrv.ko...done. Loaded symbols for /boot/modules/vboxdrv.ko Reading symbols from /boot/modules/nvidia.ko...done. Loaded symbols for /boot/modules/nvidia.ko Reading symbols from /boot/kernel/ipmi.ko.symbols...done. Loaded symbols for /boot/kernel/ipmi.ko.symbols Reading symbols from /boot/kernel/ipmi_linux.ko.symbols...done. Loaded symbols for /boot/kernel/ipmi_linux.ko.symbols Reading symbols from /boot/kernel/radeonkms.ko.symbols...done. Loaded symbols for /boot/kernel/radeonkms.ko.symbols Reading symbols from /boot/kernel/iicbb.ko.symbols...done. Loaded symbols for /boot/kernel/iicbb.ko.symbols Reading symbols from /boot/kernel/iicbus.ko.symbols...done. Loaded symbols for /boot/kernel/iicbus.ko.symbols Reading symbols from /boot/kernel/iic.ko.symbols...done. Loaded symbols for /boot/kernel/iic.ko.symbols Reading symbols from /boot/kernel/d
