Re: amd64 head -r329465 (non-debug build, but with symbols): "panic: spin lock held too long" during make check-old, reported during a sys_vfork
Le 20/02/2018 à 21:20, Mateusz Guzik a écrit : Committed in https://svnweb.freebsd.org/base?view=revision&revision=329660 thanks for reporting Thanks Mateusz, Konstantin, issue solved. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: amd64 head -r329465 (non-debug build, but with symbols): "panic: spin lock held too long" during make check-old, reported during a sys_vfork
Committed in https://svnweb.freebsd.org/base?view=revision&revision=329660 thanks for reporting On Tue, Feb 20, 2018 at 6:06 PM, Mateusz Guzik wrote: > I missed a consumer, try this: > > diff --git a/sys/kern/sys_procdesc.c b/sys/kern/sys_procdesc.c > index 5e8928cb1534..174fffc5c666 100644 > --- a/sys/kern/sys_procdesc.c > +++ b/sys/kern/sys_procdesc.c > @@ -398,7 +398,6 @@ procdesc_close(struct file *fp, struct thread *td) > * process's reference to the process descriptor > when it > * calls back into procdesc_reap(). > */ > - PROC_SLOCK(p); > proc_reap(curthread, p, NULL, 0); > } else { > /* > > > On Tue, Feb 20, 2018 at 5:50 PM, Juan Ramón Molina Menor > wrote: > >> I committed the fix in >>> https://svnweb.freebsd.org/base?view=revision&revision=329542 >>> >>> i.e. should be stable from this point on. >>> >> >> Hi! >> >> It is maybe unrelated, but recent commits have broken my system with a >> similar error. I did not have panics with a system built around December, >> but since updating first to r329555 then today to r329641 I’m getting a >> reproducible panic when logging out from a Lumina desktop session: >> >> Unread portion of the kernel message buffer: >> spin lock 0xf8000d440020 (process slock) held by 0xf8000daed560 >> (tid 100111) too long >> panic: spin lock held too long >> cpuid = 1 >> time = 1519143505 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame >> 0xfe5c15e0 >> vpanic() at vpanic+0x18d/frame 0xfe5c1640 >> panic() at panic+0x43/frame 0xfe5c16a0 >> _mtx_lock_indefinite_check() at _mtx_lock_indefinite_check+0x71/frame >> 0xfe5c16b0 >> mtx_spin_wait_unlocked() at mtx_spin_wait_unlocked+0x59/frame >> 0xfe5c16e0 >> proc_reap() at proc_reap+0x24/frame 0xfe5c1720 >> procdesc_close() at procdesc_close+0x125/frame 0xfe5c1760 >> closef() at closef+0x251/frame 0xfe5c17f0 >> fdescfree_fds() at fdescfree_fds+0x90/frame 0xfe5c1840 >> fdescfree() at fdescfree+0x4df/frame 0xfe5c1900 >> exit1() at exit1+0x508/frame 0xfe5c1970 >> sys_sys_exit() at sys_sys_exit+0xd/frame 0xfe5c1980 >> amd64_syscall() at amd64_syscall+0xa48/frame 0xfe5c1ab0 >> fast_syscall_common() at fast_syscall_common+0x101/frame 0x7fffea90 >> Uptime: 17m45s >> Dumping 327 out of 3990 MB:..5%..15%..25%..35%..44%..5 >> 4%..64%..74%..84%..93% >> >> Reading symbols from /boot/kernel/linux.ko...done. >> Loaded symbols for /boot/kernel/linux.ko >> Reading symbols from /boot/kernel/linux_common.ko...done. >> Loaded symbols for /boot/kernel/linux_common.ko >> Reading symbols from /boot/kernel/acpi_ibm.ko...done. >> Loaded symbols for /boot/kernel/acpi_ibm.ko >> Reading symbols from /boot/kernel/iwm7260fw.ko...done. >> Loaded symbols for /boot/kernel/iwm7260fw.ko >> Reading symbols from /boot/kernel/coretemp.ko...done. >> Loaded symbols for /boot/kernel/coretemp.ko >> Reading symbols from /boot/kernel/if_iwm.ko...done. >> Loaded symbols for /boot/kernel/if_iwm.ko >> Reading symbols from /boot/kernel/acpi_video.ko...done. >> Loaded symbols for /boot/kernel/acpi_video.ko >> Reading symbols from /boot/kernel/nullfs.ko...done. >> Loaded symbols for /boot/kernel/nullfs.ko >> Reading symbols from /boot/kernel/fdescfs.ko...done. >> Loaded symbols for /boot/kernel/fdescfs.ko >> Reading symbols from /boot/kernel/i915kms.ko...done. >> Loaded symbols for /boot/kernel/i915kms.ko >> Reading symbols from /boot/kernel/drm2.ko...done. >> Loaded symbols for /boot/kernel/drm2.ko >> Reading symbols from /boot/kernel/iicbus.ko...done. >> Loaded symbols for /boot/kernel/iicbus.ko >> Reading symbols from /boot/kernel/iic.ko...done. >> Loaded symbols for /boot/kernel/iic.ko >> Reading symbols from /boot/kernel/iicbb.ko...done. >> Loaded symbols for /boot/kernel/iicbb.ko >> #0 cpustop_handler () at /usr/src/sys/x86/x86/mp_x86.c:1324 >> 1324CPU_SET_ATOMIC(cpu, &stopped_cpus); >> (kgdb) bt >> #0 cpustop_handler () at /usr/src/sys/x86/x86/mp_x86.c:1324 >> #1 0x80e29fb4 in ipi_nmi_handler () at >> /usr/src/sys/x86/x86/mp_x86.c:1280 >> #2 0x80d09a79 in trap (frame=0x8158bef0) >> at /usr/src/sys/amd64/amd64/trap.c:188 >> #3 0x80cec054 in nmi_calltrap () at >> /usr/src/sys/amd64/amd64/exception.S:633 >> #4 0x80e1aaef in acpi_cpu_idle_mwait (mwait_hint=0) at >> cpufunc.h:611 >> Previous frame inner to this frame (corrupt stack?) >> Current language: auto; currently minimal >> >> kgdb is over my head, but I can provide more details under some guidance. >> >> Hope it helps, >> Juan >> >> > > > -- > Mateusz Guzik > -- Mateusz Guzik ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe,
Re: amd64 head -r329465 (non-debug build, but with symbols): "panic: spin lock held too long" during make check-old, reported during a sys_vfork
On Tue, Feb 20, 2018 at 05:50:57PM +0100, Juan Ram?n Molina Menor wrote: > > I committed the fix in > > https://svnweb.freebsd.org/base?view=revision&revision=329542 > > > > i.e. should be stable from this point on. > > Hi! > > It is maybe unrelated, but recent commits have broken my system with a > similar error. I did not have panics with a system built around > December, but since updating first to r329555 then today to r329641 I?m > getting a reproducible panic when logging out from a Lumina desktop session: > > Unread portion of the kernel message buffer: > spin lock 0xf8000d440020 (process slock) held by 0xf8000daed560 > (tid 100111) too long > panic: spin lock held too long > cpuid = 1 > time = 1519143505 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame > 0xfe5c15e0 > vpanic() at vpanic+0x18d/frame 0xfe5c1640 > panic() at panic+0x43/frame 0xfe5c16a0 > _mtx_lock_indefinite_check() at _mtx_lock_indefinite_check+0x71/frame > 0xfe5c16b0 > mtx_spin_wait_unlocked() at mtx_spin_wait_unlocked+0x59/frame > 0xfe5c16e0 > proc_reap() at proc_reap+0x24/frame 0xfe5c1720 > procdesc_close() at procdesc_close+0x125/frame 0xfe5c1760 > closef() at closef+0x251/frame 0xfe5c17f0 > fdescfree_fds() at fdescfree_fds+0x90/frame 0xfe5c1840 > fdescfree() at fdescfree+0x4df/frame 0xfe5c1900 > exit1() at exit1+0x508/frame 0xfe5c1970 > sys_sys_exit() at sys_sys_exit+0xd/frame 0xfe5c1980 > amd64_syscall() at amd64_syscall+0xa48/frame 0xfe5c1ab0 > fast_syscall_common() at fast_syscall_common+0x101/frame 0x7fffea90 > Uptime: 17m45s > Dumping 327 out of 3990 MB:..5%..15%..25%..35%..44%..54%..64%..74%..84%..93% > > Reading symbols from /boot/kernel/linux.ko...done. > Loaded symbols for /boot/kernel/linux.ko > Reading symbols from /boot/kernel/linux_common.ko...done. > Loaded symbols for /boot/kernel/linux_common.ko > Reading symbols from /boot/kernel/acpi_ibm.ko...done. > Loaded symbols for /boot/kernel/acpi_ibm.ko > Reading symbols from /boot/kernel/iwm7260fw.ko...done. > Loaded symbols for /boot/kernel/iwm7260fw.ko > Reading symbols from /boot/kernel/coretemp.ko...done. > Loaded symbols for /boot/kernel/coretemp.ko > Reading symbols from /boot/kernel/if_iwm.ko...done. > Loaded symbols for /boot/kernel/if_iwm.ko > Reading symbols from /boot/kernel/acpi_video.ko...done. > Loaded symbols for /boot/kernel/acpi_video.ko > Reading symbols from /boot/kernel/nullfs.ko...done. > Loaded symbols for /boot/kernel/nullfs.ko > Reading symbols from /boot/kernel/fdescfs.ko...done. > Loaded symbols for /boot/kernel/fdescfs.ko > Reading symbols from /boot/kernel/i915kms.ko...done. > Loaded symbols for /boot/kernel/i915kms.ko > Reading symbols from /boot/kernel/drm2.ko...done. > Loaded symbols for /boot/kernel/drm2.ko > Reading symbols from /boot/kernel/iicbus.ko...done. > Loaded symbols for /boot/kernel/iicbus.ko > Reading symbols from /boot/kernel/iic.ko...done. > Loaded symbols for /boot/kernel/iic.ko > Reading symbols from /boot/kernel/iicbb.ko...done. > Loaded symbols for /boot/kernel/iicbb.ko > #0 cpustop_handler () at /usr/src/sys/x86/x86/mp_x86.c:1324 > 1324 CPU_SET_ATOMIC(cpu, &stopped_cpus); > (kgdb) bt > #0 cpustop_handler () at /usr/src/sys/x86/x86/mp_x86.c:1324 > #1 0x80e29fb4 in ipi_nmi_handler () at > /usr/src/sys/x86/x86/mp_x86.c:1280 > #2 0x80d09a79 in trap (frame=0x8158bef0) > at /usr/src/sys/amd64/amd64/trap.c:188 > #3 0x80cec054 in nmi_calltrap () at > /usr/src/sys/amd64/amd64/exception.S:633 > #4 0x80e1aaef in acpi_cpu_idle_mwait (mwait_hint=0) at > cpufunc.h:611 > Previous frame inner to this frame (corrupt stack?) > Current language: auto; currently minimal > > kgdb is over my head, but I can provide more details under some guidance. Use this. diff --git a/sys/kern/sys_procdesc.c b/sys/kern/sys_procdesc.c index 5e8928cb153..174fffc5c66 100644 --- a/sys/kern/sys_procdesc.c +++ b/sys/kern/sys_procdesc.c @@ -398,7 +398,6 @@ procdesc_close(struct file *fp, struct thread *td) * process's reference to the process descriptor when it * calls back into procdesc_reap(). */ - PROC_SLOCK(p); proc_reap(curthread, p, NULL, 0); } else { /* ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: amd64 head -r329465 (non-debug build, but with symbols): "panic: spin lock held too long" during make check-old, reported during a sys_vfork
I missed a consumer, try this: diff --git a/sys/kern/sys_procdesc.c b/sys/kern/sys_procdesc.c index 5e8928cb1534..174fffc5c666 100644 --- a/sys/kern/sys_procdesc.c +++ b/sys/kern/sys_procdesc.c @@ -398,7 +398,6 @@ procdesc_close(struct file *fp, struct thread *td) * process's reference to the process descriptor when it * calls back into procdesc_reap(). */ - PROC_SLOCK(p); proc_reap(curthread, p, NULL, 0); } else { /* On Tue, Feb 20, 2018 at 5:50 PM, Juan Ramón Molina Menor wrote: > I committed the fix in >> https://svnweb.freebsd.org/base?view=revision&revision=329542 >> >> i.e. should be stable from this point on. >> > > Hi! > > It is maybe unrelated, but recent commits have broken my system with a > similar error. I did not have panics with a system built around December, > but since updating first to r329555 then today to r329641 I’m getting a > reproducible panic when logging out from a Lumina desktop session: > > Unread portion of the kernel message buffer: > spin lock 0xf8000d440020 (process slock) held by 0xf8000daed560 > (tid 100111) too long > panic: spin lock held too long > cpuid = 1 > time = 1519143505 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame > 0xfe5c15e0 > vpanic() at vpanic+0x18d/frame 0xfe5c1640 > panic() at panic+0x43/frame 0xfe5c16a0 > _mtx_lock_indefinite_check() at _mtx_lock_indefinite_check+0x71/frame > 0xfe5c16b0 > mtx_spin_wait_unlocked() at mtx_spin_wait_unlocked+0x59/frame > 0xfe5c16e0 > proc_reap() at proc_reap+0x24/frame 0xfe5c1720 > procdesc_close() at procdesc_close+0x125/frame 0xfe5c1760 > closef() at closef+0x251/frame 0xfe5c17f0 > fdescfree_fds() at fdescfree_fds+0x90/frame 0xfe5c1840 > fdescfree() at fdescfree+0x4df/frame 0xfe5c1900 > exit1() at exit1+0x508/frame 0xfe5c1970 > sys_sys_exit() at sys_sys_exit+0xd/frame 0xfe5c1980 > amd64_syscall() at amd64_syscall+0xa48/frame 0xfe5c1ab0 > fast_syscall_common() at fast_syscall_common+0x101/frame 0x7fffea90 > Uptime: 17m45s > Dumping 327 out of 3990 MB:..5%..15%..25%..35%..44%..5 > 4%..64%..74%..84%..93% > > Reading symbols from /boot/kernel/linux.ko...done. > Loaded symbols for /boot/kernel/linux.ko > Reading symbols from /boot/kernel/linux_common.ko...done. > Loaded symbols for /boot/kernel/linux_common.ko > Reading symbols from /boot/kernel/acpi_ibm.ko...done. > Loaded symbols for /boot/kernel/acpi_ibm.ko > Reading symbols from /boot/kernel/iwm7260fw.ko...done. > Loaded symbols for /boot/kernel/iwm7260fw.ko > Reading symbols from /boot/kernel/coretemp.ko...done. > Loaded symbols for /boot/kernel/coretemp.ko > Reading symbols from /boot/kernel/if_iwm.ko...done. > Loaded symbols for /boot/kernel/if_iwm.ko > Reading symbols from /boot/kernel/acpi_video.ko...done. > Loaded symbols for /boot/kernel/acpi_video.ko > Reading symbols from /boot/kernel/nullfs.ko...done. > Loaded symbols for /boot/kernel/nullfs.ko > Reading symbols from /boot/kernel/fdescfs.ko...done. > Loaded symbols for /boot/kernel/fdescfs.ko > Reading symbols from /boot/kernel/i915kms.ko...done. > Loaded symbols for /boot/kernel/i915kms.ko > Reading symbols from /boot/kernel/drm2.ko...done. > Loaded symbols for /boot/kernel/drm2.ko > Reading symbols from /boot/kernel/iicbus.ko...done. > Loaded symbols for /boot/kernel/iicbus.ko > Reading symbols from /boot/kernel/iic.ko...done. > Loaded symbols for /boot/kernel/iic.ko > Reading symbols from /boot/kernel/iicbb.ko...done. > Loaded symbols for /boot/kernel/iicbb.ko > #0 cpustop_handler () at /usr/src/sys/x86/x86/mp_x86.c:1324 > 1324CPU_SET_ATOMIC(cpu, &stopped_cpus); > (kgdb) bt > #0 cpustop_handler () at /usr/src/sys/x86/x86/mp_x86.c:1324 > #1 0x80e29fb4 in ipi_nmi_handler () at > /usr/src/sys/x86/x86/mp_x86.c:1280 > #2 0x80d09a79 in trap (frame=0x8158bef0) > at /usr/src/sys/amd64/amd64/trap.c:188 > #3 0x80cec054 in nmi_calltrap () at /usr/src/sys/amd64/amd64/excep > tion.S:633 > #4 0x80e1aaef in acpi_cpu_idle_mwait (mwait_hint=0) at > cpufunc.h:611 > Previous frame inner to this frame (corrupt stack?) > Current language: auto; currently minimal > > kgdb is over my head, but I can provide more details under some guidance. > > Hope it helps, > Juan > > -- Mateusz Guzik ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: amd64 head -r329465 (non-debug build, but with symbols): "panic: spin lock held too long" during make check-old, reported during a sys_vfork
I committed the fix in https://svnweb.freebsd.org/base?view=revision&revision=329542 i.e. should be stable from this point on. Hi! It is maybe unrelated, but recent commits have broken my system with a similar error. I did not have panics with a system built around December, but since updating first to r329555 then today to r329641 I’m getting a reproducible panic when logging out from a Lumina desktop session: Unread portion of the kernel message buffer: spin lock 0xf8000d440020 (process slock) held by 0xf8000daed560 (tid 100111) too long panic: spin lock held too long cpuid = 1 time = 1519143505 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfe5c15e0 vpanic() at vpanic+0x18d/frame 0xfe5c1640 panic() at panic+0x43/frame 0xfe5c16a0 _mtx_lock_indefinite_check() at _mtx_lock_indefinite_check+0x71/frame 0xfe5c16b0 mtx_spin_wait_unlocked() at mtx_spin_wait_unlocked+0x59/frame 0xfe5c16e0 proc_reap() at proc_reap+0x24/frame 0xfe5c1720 procdesc_close() at procdesc_close+0x125/frame 0xfe5c1760 closef() at closef+0x251/frame 0xfe5c17f0 fdescfree_fds() at fdescfree_fds+0x90/frame 0xfe5c1840 fdescfree() at fdescfree+0x4df/frame 0xfe5c1900 exit1() at exit1+0x508/frame 0xfe5c1970 sys_sys_exit() at sys_sys_exit+0xd/frame 0xfe5c1980 amd64_syscall() at amd64_syscall+0xa48/frame 0xfe5c1ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0x7fffea90 Uptime: 17m45s Dumping 327 out of 3990 MB:..5%..15%..25%..35%..44%..54%..64%..74%..84%..93% Reading symbols from /boot/kernel/linux.ko...done. Loaded symbols for /boot/kernel/linux.ko Reading symbols from /boot/kernel/linux_common.ko...done. Loaded symbols for /boot/kernel/linux_common.ko Reading symbols from /boot/kernel/acpi_ibm.ko...done. Loaded symbols for /boot/kernel/acpi_ibm.ko Reading symbols from /boot/kernel/iwm7260fw.ko...done. Loaded symbols for /boot/kernel/iwm7260fw.ko Reading symbols from /boot/kernel/coretemp.ko...done. Loaded symbols for /boot/kernel/coretemp.ko Reading symbols from /boot/kernel/if_iwm.ko...done. Loaded symbols for /boot/kernel/if_iwm.ko Reading symbols from /boot/kernel/acpi_video.ko...done. Loaded symbols for /boot/kernel/acpi_video.ko Reading symbols from /boot/kernel/nullfs.ko...done. Loaded symbols for /boot/kernel/nullfs.ko Reading symbols from /boot/kernel/fdescfs.ko...done. Loaded symbols for /boot/kernel/fdescfs.ko Reading symbols from /boot/kernel/i915kms.ko...done. Loaded symbols for /boot/kernel/i915kms.ko Reading symbols from /boot/kernel/drm2.ko...done. Loaded symbols for /boot/kernel/drm2.ko Reading symbols from /boot/kernel/iicbus.ko...done. Loaded symbols for /boot/kernel/iicbus.ko Reading symbols from /boot/kernel/iic.ko...done. Loaded symbols for /boot/kernel/iic.ko Reading symbols from /boot/kernel/iicbb.ko...done. Loaded symbols for /boot/kernel/iicbb.ko #0 cpustop_handler () at /usr/src/sys/x86/x86/mp_x86.c:1324 1324 CPU_SET_ATOMIC(cpu, &stopped_cpus); (kgdb) bt #0 cpustop_handler () at /usr/src/sys/x86/x86/mp_x86.c:1324 #1 0x80e29fb4 in ipi_nmi_handler () at /usr/src/sys/x86/x86/mp_x86.c:1280 #2 0x80d09a79 in trap (frame=0x8158bef0) at /usr/src/sys/amd64/amd64/trap.c:188 #3 0x80cec054 in nmi_calltrap () at /usr/src/sys/amd64/amd64/exception.S:633 #4 0x80e1aaef in acpi_cpu_idle_mwait (mwait_hint=0) at cpufunc.h:611 Previous frame inner to this frame (corrupt stack?) Current language: auto; currently minimal kgdb is over my head, but I can provide more details under some guidance. Hope it helps, Juan ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: amd64 head -r329465 (non-debug build, but with symbols): "panic: spin lock held too long" during make check-old, reported during a sys_vfork
On 2018-Feb-18, at 4:55 PM, Mateusz Guzik wrote: > I committed the fix in > https://svnweb.freebsd.org/base?view=revision&revision=329542 > > i.e. should be stable from this point on. Thanks! === Mark Millard marklmi at yahoo.com ( markmi at dsl-only.net is going away in 2018-Feb, late) ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: amd64 head -r329465 (non-debug build, but with symbols): "panic: spin lock held too long" during make check-old, reported during a sys_vfork
I committed the fix in https://svnweb.freebsd.org/base?view=revision&revision=329542 i.e. should be stable from this point on. On Sun, Feb 18, 2018 at 11:55 PM, Mateusz Guzik wrote: > On Sun, Feb 18, 2018 at 11:24 PM, Trond Endrestøl < > trond.endres...@fagskolen.gjovik.no> wrote: > >> On Sun, 18 Feb 2018 22:33+0100, Mateusz Guzik wrote: >> >> > On Sun, Feb 18, 2018 at 9:38 PM, Trond Endrestøl < >> > trond.endres...@fagskolen.gjovik.no> wrote: >> > >> > > On Sun, 18 Feb 2018 11:51-0800, Mark Millard wrote: >> > > >> > > > Note: -r329448 was reverted in -r329461 : racy. >> > > >> > > True. I got a crash when compiling r329451 while running r329449. >> > > I've now booted the r329422 ZFS BE and I'm attempting to build >> > > r329529. >> > > >> > >> > Looking around strongly suggests r329448 is the culprit. If you can >> verify >> > 329447 works fine we are mostly done here. >> >> I noticed no errors in r329447. When r329529 is built and installed, >> I'll try to incrementally build and install r329531. >> > > Can you grab a panicking kernel and apply this: > https://people.freebsd.org/~mjg/wait_unlocked.diff > > there may be debug printfs signifying the problem condition was hit, > however the patch should fix the panic > > -- > Mateusz Guzik > -- Mateusz Guzik ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: amd64 head -r329465 (non-debug build, but with symbols): "panic: spin lock held too long" during make check-old, reported during a sys_vfork
On Sun, Feb 18, 2018 at 11:24 PM, Trond Endrestøl < trond.endres...@fagskolen.gjovik.no> wrote: > On Sun, 18 Feb 2018 22:33+0100, Mateusz Guzik wrote: > > > On Sun, Feb 18, 2018 at 9:38 PM, Trond Endrestøl < > > trond.endres...@fagskolen.gjovik.no> wrote: > > > > > On Sun, 18 Feb 2018 11:51-0800, Mark Millard wrote: > > > > > > > Note: -r329448 was reverted in -r329461 : racy. > > > > > > True. I got a crash when compiling r329451 while running r329449. > > > I've now booted the r329422 ZFS BE and I'm attempting to build > > > r329529. > > > > > > > Looking around strongly suggests r329448 is the culprit. If you can > verify > > 329447 works fine we are mostly done here. > > I noticed no errors in r329447. When r329529 is built and installed, > I'll try to incrementally build and install r329531. > Can you grab a panicking kernel and apply this: https://people.freebsd.org/~mjg/wait_unlocked.diff there may be debug printfs signifying the problem condition was hit, however the patch should fix the panic -- Mateusz Guzik ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: amd64 head -r329465 (non-debug build, but with symbols): "panic: spin lock held too long" during make check-old, reported during a sys_vfork
On Sun, 18 Feb 2018 22:33+0100, Mateusz Guzik wrote: > On Sun, Feb 18, 2018 at 9:38 PM, Trond Endrestøl < > trond.endres...@fagskolen.gjovik.no> wrote: > > > On Sun, 18 Feb 2018 11:51-0800, Mark Millard wrote: > > > > > Note: -r329448 was reverted in -r329461 : racy. > > > > True. I got a crash when compiling r329451 while running r329449. > > I've now booted the r329422 ZFS BE and I'm attempting to build > > r329529. > > > > Looking around strongly suggests r329448 is the culprit. If you can verify > 329447 works fine we are mostly done here. I noticed no errors in r329447. When r329529 is built and installed, I'll try to incrementally build and install r329531. > Note the revision got reverted and different variant got in in r329531. > > That said, if r329447 works then the issue should be already fixed and in > particular fresh head should work fine. -- Trond. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: amd64 head -r329465 (non-debug build, but with symbols): "panic: spin lock held too long" during make check-old, reported during a sys_vfork
On Sun, Feb 18, 2018 at 10:50 PM, Mark Millard wrote: > > > On 2018-Feb-18, at 1:46 PM, Mark Millard wrote: > > > On 2018-Feb-18, at 1:33 PM, Mateusz Guzik wrote: > > > >> On Sun, Feb 18, 2018 at 9:38 PM, Trond Endrestøl < > >> trond.endres...@fagskolen.gjovik.no> wrote: > >> > >>> On Sun, 18 Feb 2018 11:51-0800, Mark Millard wrote: > >>> > Note: -r329448 was reverted in -r329461 : racy. > >>> > >>> True. I got a crash when compiling r329451 while running r329449. > >>> I've now booted the r329422 ZFS BE and I'm attempting to build > >>> r329529. > >>> > >> > >> Looking around strongly suggests r329448 is the culprit. If you can > verify > >> 329447 works fine we are mostly done here. > >> > >> Note the revision got reverted and different variant got in in r329531. > >> > >> That said, if r329447 works then the issue should be already fixed and > in > >> particular fresh head should work fine. > > > > My initial problem was with -r329465, which is after -r329461 reverted > > -r329488 . Trond reported in one note that he had problems with > > -r329464 , also after -r329488 was reverted. Trond has also reported > > -r329449 failed. > > Dumb typos above: I meant -r329448 instead of -r329488 both times. > > Ok, I think I see the bug: exit1 does: PROC_SLOCK(p); p->p_state = PRS_ZOMBIE; /* work continues */ pre-patch proc_to_reap does an equivalent of: if (p->p_state == PRS_ZOMBIE) { PROC_SLOCK(p); PROC_SUNLOCK(p); reap; } It is possible the exiting thread will be caught just after setting the state to PRS_ZOMBIE. With the slock/sunlock cycle we guarantee the reaping thread will wait for it to finish. Without the cycle we can end up reaping the still exiting thread. I'll fix it soon(tm). -- Mateusz Guzik ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: amd64 head -r329465 (non-debug build, but with symbols): "panic: spin lock held too long" during make check-old, reported during a sys_vfork
On 2018-Feb-18, at 1:33 PM, Mateusz Guzik wrote: > On Sun, Feb 18, 2018 at 9:38 PM, Trond Endrestøl < > trond.endres...@fagskolen.gjovik.no> wrote: > >> On Sun, 18 Feb 2018 11:51-0800, Mark Millard wrote: >> >>> Note: -r329448 was reverted in -r329461 : racy. >> >> True. I got a crash when compiling r329451 while running r329449. >> I've now booted the r329422 ZFS BE and I'm attempting to build >> r329529. >> > > Looking around strongly suggests r329448 is the culprit. If you can verify > 329447 works fine we are mostly done here. > > Note the revision got reverted and different variant got in in r329531. > > That said, if r329447 works then the issue should be already fixed and in > particular fresh head should work fine. My initial problem was with -r329465, which is after -r329461 reverted -r329488 . Trond reported in one note that he had problems with -r329464 , also after -r329488 was reverted. Trond has also reported -r329449 failed. I did manage to revert to -r329447 earlier and so far the results suggests that it works. From this I get that -r329449 is the the one that is common to all the so--far failing combinations. -r329448 is not common to all of them. === Mark Millard marklmi at yahoo.com ( markmi at dsl-only.net is going away in 2018-Feb, late) ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: amd64 head -r329465 (non-debug build, but with symbols): "panic: spin lock held too long" during make check-old, reported during a sys_vfork
On 2018-Feb-18, at 1:46 PM, Mark Millard wrote: > On 2018-Feb-18, at 1:33 PM, Mateusz Guzik wrote: > >> On Sun, Feb 18, 2018 at 9:38 PM, Trond Endrestøl < >> trond.endres...@fagskolen.gjovik.no> wrote: >> >>> On Sun, 18 Feb 2018 11:51-0800, Mark Millard wrote: >>> Note: -r329448 was reverted in -r329461 : racy. >>> >>> True. I got a crash when compiling r329451 while running r329449. >>> I've now booted the r329422 ZFS BE and I'm attempting to build >>> r329529. >>> >> >> Looking around strongly suggests r329448 is the culprit. If you can verify >> 329447 works fine we are mostly done here. >> >> Note the revision got reverted and different variant got in in r329531. >> >> That said, if r329447 works then the issue should be already fixed and in >> particular fresh head should work fine. > > My initial problem was with -r329465, which is after -r329461 reverted > -r329488 . Trond reported in one note that he had problems with > -r329464 , also after -r329488 was reverted. Trond has also reported > -r329449 failed. Dumb typos above: I meant -r329448 instead of -r329488 both times. > I did manage to revert to -r329447 earlier and so far the results > suggests that it works. > > From this I get that -r329449 is the the one that is common to > all the so--far failing combinations. -r329448 is not common to > all of them. === Mark Millard marklmi at yahoo.com ( markmi at dsl-only.net is going away in 2018-Feb, late) ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: amd64 head -r329465 (non-debug build, but with symbols): "panic: spin lock held too long" during make check-old, reported during a sys_vfork
On Sun, Feb 18, 2018 at 9:38 PM, Trond Endrestøl < trond.endres...@fagskolen.gjovik.no> wrote: > On Sun, 18 Feb 2018 11:51-0800, Mark Millard wrote: > > > Note: -r329448 was reverted in -r329461 : racy. > > True. I got a crash when compiling r329451 while running r329449. > I've now booted the r329422 ZFS BE and I'm attempting to build > r329529. > Looking around strongly suggests r329448 is the culprit. If you can verify 329447 works fine we are mostly done here. Note the revision got reverted and different variant got in in r329531. That said, if r329447 works then the issue should be already fixed and in particular fresh head should work fine. -- Mateusz Guzik ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: amd64 head -r329465 (non-debug build, but with symbols): "panic: spin lock held too long" during make check-old, reported during a sys_vfork
On Sun, 18 Feb 2018 11:51-0800, Mark Millard wrote: > Note: -r329448 was reverted in -r329461 : racy. True. I got a crash when compiling r329451 while running r329449. I've now booted the r329422 ZFS BE and I'm attempting to build r329529. -- Trond. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: amd64 head -r329465 (non-debug build, but with symbols): "panic: spin lock held too long" during make check-old, reported during a sys_vfork
On 2018-Feb-18, at 10:08 AM, Mateusz Guzik wrote: > Can you please bisect this? There is another report stating that r329418 > works fine. I saw that Trond indicated an intent to test -r329418 but I've not seen any reports about -r329418 or how much activity was used to make any judgment about its status. But I can assume -r329418 is good if you want. Bisecting is likely going to be problematical for self-updates: builds and installs and such can crash, making the installs risky. I do not have an alternate builder for amd64 set up. Even without that, it is not clear how many hours of build-related activity it takes to have a high probability that the problem is gone. (I've seen widely variable amounts of activity between failures in -r329465 .) It is obvious to try an earlier version after failure but not obvious when to try a later version. My FreeBSD time is also rather limited (compared to historically over the last few years), so the activity could be spread over parts of various weekends, depending on how it goes. >> On Sun, Feb 18, 2018 at 6:35 PM, Mark Millard >> wrote: >> >> On 2018-Feb-17, at 6:10 PM, Mark Millard wrote: >> >> > [Some more information added, from /usr/libexec/kgdb use.] >> > >> > On 2018-Feb-17, at 5:39 PM, Mark Millard >> > wrote: >> > >> >> This is for FreeBSD running under Hyper-V on a Windows 10 Pro machine. >> >> The FreeBSD "disk" bindings are to SSDs, not the insides of NTFS files. >> >> 29 logical processors assigned to FreeBSD (on a 32-thread Ryzen >> >> Threadripper 1950X). No other Hyper-V use. >> >> Trond's report seems to be for a "4 core" Intel i7 context (as seen >> by FreeBSD in virtual box). So Ryzen seems to be non-essential for >> reproduction. >> >> Both of our reports are from some form of using FreeBSD in a virtual >> machine (Hyper-V and VirtualBox). I do not know if that is a required >> type of context or not. >> >> >> This happened during: >> >> >> >> # >> >> ~/sys_build_scripts.amd64-host/make_powerpc64vtsc_nodebug_clang_altbinutils-amd64-host.sh >> >> check-old >> >> DESTDIR=/usr/obj/DESTDIRs/clang-powerpc64-installworld_altbinutils >> >> Script started, output file is >> >> /root/sys_typescripts/typescript_make_powerpc64vtsc_nodebug_clang_altbinutils-amd64-host-2018-02-17:15:56:20 >> > Checking for old files >> >> >> >> I got another example but during a buildworld: >> >> >>> Deleting stale files in build tree... >> cd /usr/src; MACHINE_ARCH=powerpc64 MACHINE=powerpc CPUTYPE= >> BUILD_TOOLS_META=.NOMETA CC="cc -target powerpc64-unknown-freebsd12.0 >> --sysroot=/usr/obj/powerpc64vtsc_clang_altbinutils/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp >> -B/usr/local/powerpc64-unknown-freebsd12.0/bin/" CXX="c++ -target >> powerpc64-unknown-freebsd12.0 >> --sysroot=/usr/obj/powerpc64vtsc_clang_altbinutils/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp >> -B/usr/local/powerpc64-unknown-freebsd12.0/bin/" CPP="cpp -target >> powerpc64-unknown-freebsd12.0 >> --sysroot=/usr/obj/powerpc64vtsc_clang_altbinutils/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp >> -B/usr/local/powerpc64-unknown-freebsd12.0/bin/" >> AS="/usr/local/powerpc64-unknown-freebsd12.0/bin/as" >> AR="/usr/local/powerpc64-unknown-freebsd12.0/bin/ar" >> LD="/usr/local/powerpc64-unknown-freebsd12.0/bin/ld" LLVM_LINK="" >> NM=/usr/local/powerpc64-unknown-freebsd12.0/bin/nm >> OBJCOPY="/usr/local/powerpc64-unknown-freebsd12.0/bin/objcopy" >> RANLIB=/usr/local/powerpc64-unkno wn- >> freebsd12.0/bin/ranlib >> STRINGS=/usr/local/bin/powerpc64-unknown-freebsd12.0-strings >> SIZE="/usr/local/powerpc64-unknown-freebsd12.0/bin/size" INSTALL="sh >> /usr/src/tools/install.sh" >> PATH=/usr/obj/powerpc64vtsc_clang_altbinutils/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp/legacy/usr/sbin:/usr/obj/powerpc64vtsc_clang_altbinutils/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp/legacy/usr/bin:/usr/obj/powerpc64vtsc_clang_altbinutils/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp/legacy/bin:/usr/obj/powerpc64vtsc_clang_altbinutils/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp/usr/sbin:/usr/obj/powerpc64vtsc_clang_altbinutils/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp/usr/bin:/sbin:/bin:/usr/sbin:/usr/bin >> >> SYSROOT=/usr/obj/powerpc64vtsc_clang_altbinutils/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp >> make -f Makefile.inc1 BWPHASE=worldtmp >> DESTDIR=/usr/obj/powerpc64vtsc_clang_altbinutils/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp >> -DBATCH_DELETE_OLD_FILES delete-ol d d >> elete-old-libs >/dev/null >> >> load: 0.68 cmd: make 62180 [select] 25.15r 0.00u 0.00s 0% 1468k >> make: Working in: >> /usr/obj/powerpc64vtsc_clang_altbinutils/powerpc.powerpc64/usr/src/powerpc.powerpc64 >> packet_write_wait: Connection to 192.168.1.165 port 22: Broken pipe >> >> >> (I noticed the long pause and got the ^T in before the panic.) >> >> Yet again it is xargs related fork activity that gets the problem (from >> core.txt.1 ): >>
Re: amd64 head -r329465 (non-debug build, but with symbols): "panic: spin lock held too long" during make check-old, reported during a sys_vfork
On 2018-Feb-18, at 11:30 AM, Trond Endrestøl wrote: > On Sun, 18 Feb 2018 19:08+0100, Mateusz Guzik wrote: > >> Can you please bisect this? There is another report stating that r329418 >> works fine. > > My problems started yesterday with r329464. I decided to go back to > r329101 (ZFS BE), update the source tree, move forward to the latest > revision, and so on. I even emptied /usr/obj and /var/cache/ccache and > set WITHOUT_SYSTEM_COMPILER=yes in /etc/src.conf to get rid of any > bias. > > I have tried with success r329418, r329419, r329420, and r329422. > > I'm now at r329448 and have not seen any spin lock problems so far. Note: -r329448 was reverted in -r329461 : racy. > Sooner or later I'll reach r329464 and by then it should be clear > which revision is the likely culprit. === Mark Millard marklmi at yahoo.com ( markmi at dsl-only.net is going away in 2018-Feb, late) ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: amd64 head -r329465 (non-debug build, but with symbols): "panic: spin lock held too long" during make check-old, reported during a sys_vfork
On Sun, 18 Feb 2018 19:08+0100, Mateusz Guzik wrote: > Can you please bisect this? There is another report stating that r329418 > works fine. My problems started yesterday with r329464. I decided to go back to r329101 (ZFS BE), update the source tree, move forward to the latest revision, and so on. I even emptied /usr/obj and /var/cache/ccache and set WITHOUT_SYSTEM_COMPILER=yes in /etc/src.conf to get rid of any bias. I have tried with success r329418, r329419, r329420, and r329422. I'm now at r329448 and have not seen any spin lock problems so far. Sooner or later I'll reach r329464 and by then it should be clear which revision is the likely culprit. -- Trond. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: amd64 head -r329465 (non-debug build, but with symbols): "panic: spin lock held too long" during make check-old, reported during a sys_vfork
Can you please bisect this? There is another report stating that r329418 works fine. On Sun, Feb 18, 2018 at 6:35 PM, Mark Millard wrote: > > On 2018-Feb-17, at 6:10 PM, Mark Millard > wrote: > > > [Some more information added, from /usr/libexec/kgdb use.] > > > > On 2018-Feb-17, at 5:39 PM, Mark Millard > wrote: > > > >> This is for FreeBSD running under Hyper-V on a Windows 10 Pro machine. > >> The FreeBSD "disk" bindings are to SSDs, not the insides of NTFS files. > >> 29 logical processors assigned to FreeBSD (on a 32-thread Ryzen > >> Threadripper 1950X). No other Hyper-V use. > > Trond's report seems to be for a "4 core" Intel i7 context (as seen > by FreeBSD in virtual box). So Ryzen seems to be non-essential for > reproduction. > > Both of our reports are from some form of using FreeBSD in a virtual > machine (Hyper-V and VirtualBox). I do not know if that is a required > type of context or not. > > >> This happened during: > >> > >> # ~/sys_build_scripts.amd64-host/make_powerpc64vtsc_ > nodebug_clang_altbinutils-amd64-host.sh check-old > DESTDIR=/usr/obj/DESTDIRs/clang-powerpc64-installworld_altbinutils > >> Script started, output file is /root/sys_typescripts/ > typescript_make_powerpc64vtsc_nodebug_clang_altbinutils- > amd64-host-2018-02-17:15:56:20 > > Checking for old files > >> > > I got another example but during a buildworld: > > >>> Deleting stale files in build tree... > cd /usr/src; MACHINE_ARCH=powerpc64 MACHINE=powerpc CPUTYPE= > BUILD_TOOLS_META=.NOMETA CC="cc -target powerpc64-unknown-freebsd12.0 > --sysroot=/usr/obj/powerpc64vtsc_clang_altbinutils/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp > -B/usr/local/powerpc64-unknown-freebsd12.0/bin/" CXX="c++ -target > powerpc64-unknown-freebsd12.0 --sysroot=/usr/obj/powerpc64vtsc_clang_ > altbinutils/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp > -B/usr/local/powerpc64-unknown-freebsd12.0/bin/" CPP="cpp -target > powerpc64-unknown-freebsd12.0 --sysroot=/usr/obj/powerpc64vtsc_clang_ > altbinutils/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp > -B/usr/local/powerpc64-unknown-freebsd12.0/bin/" > AS="/usr/local/powerpc64-unknown-freebsd12.0/bin/as" > AR="/usr/local/powerpc64-unknown-freebsd12.0/bin/ar" > LD="/usr/local/powerpc64-unknown-freebsd12.0/bin/ld" LLVM_LINK="" > NM=/usr/local/powerpc64-unknown-freebsd12.0/bin/nm > OBJCOPY="/usr/local/powerpc64-unknown-freebsd12.0/bin/objcopy" > RANLIB=/usr/local/powerpc64-unknown- > freebsd12.0/bin/ranlib STRINGS=/usr/local/bin/ > powerpc64-unknown-freebsd12.0-strings > SIZE="/usr/local/powerpc64-unknown-freebsd12.0/bin/size" > INSTALL="sh /usr/src/tools/install.sh" PATH=/usr/obj/powerpc64vtsc_ > clang_altbinutils/powerpc.powerpc64/usr/src/powerpc. > powerpc64/tmp/legacy/usr/sbin:/usr/obj/powerpc64vtsc_clang_ > altbinutils/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp/ > legacy/usr/bin:/usr/obj/powerpc64vtsc_clang_altbinutils/powerpc.powerpc64/ > usr/src/powerpc.powerpc64/tmp/legacy/bin:/usr/obj/powerpc64vtsc_clang_ > altbinutils/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp/ > usr/sbin:/usr/obj/powerpc64vtsc_clang_altbinutils/powerpc.powerpc64/ > usr/src/powerpc.powerpc64/tmp/usr/bin:/sbin:/bin:/usr/sbin:/usr/bin > SYSROOT=/usr/obj/powerpc64vtsc_clang_altbinutils/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp > make -f Makefile.inc1 BWPHASE=worldtmp DESTDIR=/usr/obj/ > powerpc64vtsc_clang_altbinutils/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp > -DBATCH_DELETE_OLD_FILES delete-old d > elete-old-libs >/dev/null > > load: 0.68 cmd: make 62180 [select] 25.15r 0.00u 0.00s 0% 1468k > make: Working in: /usr/obj/powerpc64vtsc_clang_ > altbinutils/powerpc.powerpc64/usr/src/powerpc.powerpc64 > packet_write_wait: Connection to 192.168.1.165 port 22: Broken pipe > > > (I noticed the long pause and got the ^T in before the panic.) > > Yet again it is xargs related fork activity that gets the problem (from > core.txt.1 ): > > 561 Thread 100836 (PID=69982: xargs) fork_trampoline () at > /usr/src/sys/amd64/amd64/exception.S:840 > . . . > * 559 Thread 100811 (PID=62304: xargs) doadump (textdump=-2122191464) at > pcpu.h:230 > > spin lock 0x81b3cf00 (sched lock 24) held by 0xf806aa6d5000 > (tid 100836) too long > panic: spin lock held too long > cpuid = 24 > time = 1518974055 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame > 0xfe00f11304d0 > vpanic() at vpanic+0x18d/frame 0xfe00f1130530 > panic() at panic+0x43/frame 0xfe00f1130590 > _mtx_lock_indefinite_check() at _mtx_lock_indefinite_check+0x71/frame > 0xfe00f11305a0 > thread_lock_flags_() at thread_lock_flags_+0xdb/frame 0xfe00f1130610 > statclock_cnt() at statclock_cnt+0xdc/frame 0xfe00f1130650 > handleevents() at handleevents+0x113/frame 0xfe00f11306a0 > timercb() at timercb+0xa9/frame 0xfe00f11306f0 > lapic_handle_timer() at lapic_handle_timer+0xa7/frame 0xfe00f1130730 > timerint_u() at timerint_u+0x96/frame 0xfe00f1130810 > thread_
Re: amd64 head -r329465 (non-debug build, but with symbols): "panic: spin lock held too long" during make check-old, reported during a sys_vfork
On 2018-Feb-17, at 6:10 PM, Mark Millard wrote: > [Some more information added, from /usr/libexec/kgdb use.] > > On 2018-Feb-17, at 5:39 PM, Mark Millard wrote: > >> This is for FreeBSD running under Hyper-V on a Windows 10 Pro machine. >> The FreeBSD "disk" bindings are to SSDs, not the insides of NTFS files. >> 29 logical processors assigned to FreeBSD (on a 32-thread Ryzen >> Threadripper 1950X). No other Hyper-V use. Trond's report seems to be for a "4 core" Intel i7 context (as seen by FreeBSD in virtual box). So Ryzen seems to be non-essential for reproduction. Both of our reports are from some form of using FreeBSD in a virtual machine (Hyper-V and VirtualBox). I do not know if that is a required type of context or not. >> This happened during: >> >> # >> ~/sys_build_scripts.amd64-host/make_powerpc64vtsc_nodebug_clang_altbinutils-amd64-host.sh >> check-old DESTDIR=/usr/obj/DESTDIRs/clang-powerpc64-installworld_altbinutils >> Script started, output file is >> /root/sys_typescripts/typescript_make_powerpc64vtsc_nodebug_clang_altbinutils-amd64-host-2018-02-17:15:56:20 > Checking for old files >> I got another example but during a buildworld: >>> Deleting stale files in build tree... cd /usr/src; MACHINE_ARCH=powerpc64 MACHINE=powerpc CPUTYPE= BUILD_TOOLS_META=.NOMETA CC="cc -target powerpc64-unknown-freebsd12.0 --sysroot=/usr/obj/powerpc64vtsc_clang_altbinutils/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp -B/usr/local/powerpc64-unknown-freebsd12.0/bin/" CXX="c++ -target powerpc64-unknown-freebsd12.0 --sysroot=/usr/obj/powerpc64vtsc_clang_altbinutils/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp -B/usr/local/powerpc64-unknown-freebsd12.0/bin/" CPP="cpp -target powerpc64-unknown-freebsd12.0 --sysroot=/usr/obj/powerpc64vtsc_clang_altbinutils/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp -B/usr/local/powerpc64-unknown-freebsd12.0/bin/" AS="/usr/local/powerpc64-unknown-freebsd12.0/bin/as" AR="/usr/local/powerpc64-unknown-freebsd12.0/bin/ar" LD="/usr/local/powerpc64-unknown-freebsd12.0/bin/ld" LLVM_LINK="" NM=/usr/local/powerpc64-unknown-freebsd12.0/bin/nm OBJCOPY="/usr/local/powerpc64-unknown-freebsd12.0/bin/objcopy" RANLIB=/usr/local/powerpc64-unknown- freebsd12.0/bin/ranlib STRINGS=/usr/local/bin/powerpc64-unknown-freebsd12.0-strings SIZE="/usr/local/powerpc64-unknown-freebsd12.0/bin/size" INSTALL="sh /usr/src/tools/install.sh" PATH=/usr/obj/powerpc64vtsc_clang_altbinutils/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp/legacy/usr/sbin:/usr/obj/powerpc64vtsc_clang_altbinutils/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp/legacy/usr/bin:/usr/obj/powerpc64vtsc_clang_altbinutils/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp/legacy/bin:/usr/obj/powerpc64vtsc_clang_altbinutils/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp/usr/sbin:/usr/obj/powerpc64vtsc_clang_altbinutils/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp/usr/bin:/sbin:/bin:/usr/sbin:/usr/bin SYSROOT=/usr/obj/powerpc64vtsc_clang_altbinutils/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp make -f Makefile.inc1 BWPHASE=worldtmp DESTDIR=/usr/obj/powerpc64vtsc_clang_altbinutils/powerpc.powerpc64/usr/src/powerpc.powerpc64/tmp -DBATCH_DELETE_OLD_FILES delete-old d elete-old-libs >/dev/null load: 0.68 cmd: make 62180 [select] 25.15r 0.00u 0.00s 0% 1468k make: Working in: /usr/obj/powerpc64vtsc_clang_altbinutils/powerpc.powerpc64/usr/src/powerpc.powerpc64 packet_write_wait: Connection to 192.168.1.165 port 22: Broken pipe (I noticed the long pause and got the ^T in before the panic.) Yet again it is xargs related fork activity that gets the problem (from core.txt.1 ): 561 Thread 100836 (PID=69982: xargs) fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:840 . . . * 559 Thread 100811 (PID=62304: xargs) doadump (textdump=-2122191464) at pcpu.h:230 spin lock 0x81b3cf00 (sched lock 24) held by 0xf806aa6d5000 (tid 100836) too long panic: spin lock held too long cpuid = 24 time = 1518974055 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfe00f11304d0 vpanic() at vpanic+0x18d/frame 0xfe00f1130530 panic() at panic+0x43/frame 0xfe00f1130590 _mtx_lock_indefinite_check() at _mtx_lock_indefinite_check+0x71/frame 0xfe00f11305a0 thread_lock_flags_() at thread_lock_flags_+0xdb/frame 0xfe00f1130610 statclock_cnt() at statclock_cnt+0xdc/frame 0xfe00f1130650 handleevents() at handleevents+0x113/frame 0xfe00f11306a0 timercb() at timercb+0xa9/frame 0xfe00f11306f0 lapic_handle_timer() at lapic_handle_timer+0xa7/frame 0xfe00f1130730 timerint_u() at timerint_u+0x96/frame 0xfe00f1130810 thread_lock_flags_() at thread_lock_flags_+0xc1/frame 0xfe00f1130880 fork1() at fork1+0x1b9f/frame 0xfe00f1130930 sys_vfork() at sys_vfork+0x4c/frame 0xfe00f1130980 amd64_syscall() at amd64_syscall+0xa48/frame 0xfe00f1130ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0x7fffc5a0
Re: amd64 head -r329465 (non-debug build, but with symbols): "panic: spin lock held too long" during make check-old, reported during a sys_vfork
On Sat, 17 Feb 2018 17:39-0800, Mark Millard wrote: > This is for FreeBSD running under Hyper-V on a Windows 10 Pro machine. > The FreeBSD "disk" bindings are to SSDs, not the insides of NTFS files. > 29 logical processors assigned to FreeBSD (on a 32-thread Ryzen > Threadripper 1950X). No other Hyper-V use. > > This happened during: > > # > ~/sys_build_scripts.amd64-host/make_powerpc64vtsc_nodebug_clang_altbinutils-amd64-host.sh > check-old DESTDIR=/usr/obj/DESTDIRs/clang-powerpc64-installworld_altbinutils > Script started, output file is > /root/sys_typescripts/typescript_make_powerpc64vtsc_nodebug_clang_altbinutils-amd64-host-2018-02-17:15:56:20 > >>> Checking for old files > > > (Hand typed from a picture of a window's content > at slighly different times, expect typos:) > > KDB: enter: panic > [thread pid 42170 tid 100752 ] > Stopped at kdb_enter+0x3b: movq $0,kdb_why > db> call doadump > Dumping 4825 out of 110559 MB: ... (omitted) ... > Dump complete > = 0 > > > (The "pid 42170" identification as the process reporting the > issue does not seem to appear in the core.txt.0 file.) > > > # ls -lTdt /var/crash/* > -rw-r--r-- 1 root wheel 100792 Feb 17 16:09:18 2018 > /var/crash/core.txt.0 > lrwxr-xr-x 1 root wheel 8 Feb 17 16:09:08 2018 > /var/crash/vmcore.last -> vmcore.0 > lrwxr-xr-x 1 root wheel 6 Feb 17 16:09:08 2018 > /var/crash/info.last -> info.0 > -rw--- 1 root wheel 5060296704 Feb 17 16:09:08 2018 /var/crash/vmcore.0 > -rw--- 1 root wheel 392 Feb 17 16:08:59 2018 /var/crash/info.0 > -rw-r--r-- 1 root wheel 2 Feb 17 16:08:59 2018 /var/crash/bounds > -rw-r--r-- 1 root wheel 5 Nov 22 04:34:36 2017 /var/crash/minfree > > >From /var/crash/core.txt.0 : > > Unread portion of the kernel message buffer: > spin lock 0x81b2dcc0 (sched lock 5) held by 0xf8011d936560 (tid > 100691) too long > panic: spin lock held too long > cpuid = 5 > time = 1518911834 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfe00f10094d0 > vpanic() at vpanic+0x18d/frame 0xfe00f1009530 > panic() at panic+0x43/frame 0xfe00f1009590 > _mtx_lock_indefinite_check() at _mtx_lock_indefinite_check+0x71/frame > 0xfe00f10095a0 > thread_lock_flags_() at thread_lock_flags_+0xdb/frame 0xfe00f1009610 > statclock_cnt() at statclock_cnt+0xdc/frame 0xfe00f1009650 > handleevents() at handleevents+0x113/frame 0xfe00f10096a0 > timercb() at timercb+0xa9/frame 0xfe00f10096f0 > lapic_handle_timer() at lapic_handle_timer+0xa7/frame 0xfe00f1009730 > timerint_u() at timerint_u+0x96/frame 0xfe00f1009810 > thread_lock_flags_() at thread_lock_flags_+0xc1/frame 0xfe00f1009880 > fork1() at fork1+0x1b9f/frame 0xfe00f1009930 > sys_vfork() at sys_vfork+0x4c/frame 0xfe00f1009980 > amd64_syscall() at amd64_syscall+0xa48/frame 0xfe00f1009ab0 > fast_syscall_common() at fast_syscall_common+0x101/frame 0x7fffcfc0 > KDB: enter: panic > > __curthread () at ./machine/pcpu.h:230 > 230 __asm("movq %%gs:%1,%0" : "=r" (td) > (kgdb) #0 __curthread () at ./machine/pcpu.h:230 > #1 doadump (textdump=-2122191464) at /usr/src/sys/kern/kern_shutdown.c:347 > #2 0x8040b42c in db_fncall_generic (addr=, >rv=, nargs=, args=) >at /usr/src/sys/ddb/db_command.c:609 > #3 db_fncall (dummy1=, dummy2=, >dummy3=, dummy4=) >at /usr/src/sys/ddb/db_command.c:657 > #4 0x8040af79 in db_command (last_cmdp=, >cmd_table=, dopager=) >at /usr/src/sys/ddb/db_command.c:481 > #5 0x8040acf4 in db_command_loop () >at /usr/src/sys/ddb/db_command.c:534 > #6 0x8040df9f in db_trap (type=, code=) >at /usr/src/sys/ddb/db_main.c:250 > #7 0x80b370e3 in kdb_trap (type=3, code=-61456, tf=) >at /usr/src/sys/kern/subr_kdb.c:697 > #8 0x80fa2c5c in trap (frame=0xfe00f1009400) >at /usr/src/sys/amd64/amd64/trap.c:547 > #9 > #10 kdb_enter (why=0x811f280b "panic", msg=) >at /usr/src/sys/kern/subr_kdb.c:479 > #11 0x80aef17a in vpanic (fmt=, ap=0xfe00f1009570) >at /usr/src/sys/kern/kern_shutdown.c:801 > #12 0x80aeefc3 in panic (fmt=0x0) >at /usr/src/sys/kern/kern_shutdown.c:739 > #13 0x80acfa31 in _mtx_lock_indefinite_check (m=, >ldap=) at /usr/src/sys/kern/kern_mutex.c:1224 > #14 0x80acfb9b in thread_lock_flags_ (td=0xf818719d1000, >opts=, file=, line=) >at /usr/src/sys/kern/kern_mutex.c:913 > #15 0x80a89d6c in statclock_cnt (cnt=1, usermode=) >at /usr/src/sys/kern/kern_clock.c:768 > #16 0x810d0003 in handleevents (now=43230207690178, fake=0) >at /usr/src/sys/kern/kern_clocksource.c:196 > #17 0x810d0709 in timercb (et=0x81c528e8 , >arg=) at /usr/src/sys/kern/kern_clocksource.c:353 > #18 0x8110dad7 in lapic_handle_timer (frame=0xfe00f1009740) >at /usr/src/sys/x86/x86/local_apic.c
Re: amd64 head -r329465 (non-debug build, but with symbols): "panic: spin lock held too long" during make check-old, reported during a sys_vfork
[Some more information added, from /usr/libexec/kgdb use.] On 2018-Feb-17, at 5:39 PM, Mark Millard wrote: > This is for FreeBSD running under Hyper-V on a Windows 10 Pro machine. > The FreeBSD "disk" bindings are to SSDs, not the insides of NTFS files. > 29 logical processors assigned to FreeBSD (on a 32-thread Ryzen > Threadripper 1950X). No other Hyper-V use. > > This happened during: > > # > ~/sys_build_scripts.amd64-host/make_powerpc64vtsc_nodebug_clang_altbinutils-amd64-host.sh > check-old DESTDIR=/usr/obj/DESTDIRs/clang-powerpc64-installworld_altbinutils > Script started, output file is > /root/sys_typescripts/typescript_make_powerpc64vtsc_nodebug_clang_altbinutils-amd64-host-2018-02-17:15:56:20 Checking for old files > > > (Hand typed from a picture of a window's content > at slighly different times, expect typos:) > > KDB: enter: panic > [thread pid 42170 tid 100752 ] > Stopped at kdb_enter+0x3b: movq $0,kdb_why > db> call doadump > Dumping 4825 out of 110559 MB: ... (omitted) ... > Dump complete > = 0 > > > (The "pid 42170" identification as the process reporting the > issue does not seem to appear in the core.txt.0 file.) > > > # ls -lTdt /var/crash/* > -rw-r--r-- 1 root wheel 100792 Feb 17 16:09:18 2018 > /var/crash/core.txt.0 > lrwxr-xr-x 1 root wheel 8 Feb 17 16:09:08 2018 > /var/crash/vmcore.last -> vmcore.0 > lrwxr-xr-x 1 root wheel 6 Feb 17 16:09:08 2018 > /var/crash/info.last -> info.0 > -rw--- 1 root wheel 5060296704 Feb 17 16:09:08 2018 /var/crash/vmcore.0 > -rw--- 1 root wheel 392 Feb 17 16:08:59 2018 /var/crash/info.0 > -rw-r--r-- 1 root wheel 2 Feb 17 16:08:59 2018 /var/crash/bounds > -rw-r--r-- 1 root wheel 5 Nov 22 04:34:36 2017 /var/crash/minfree > > From /var/crash/core.txt.0 : > > Unread portion of the kernel message buffer: > spin lock 0x81b2dcc0 (sched lock 5) held by 0xf8011d936560 (tid > 100691) too long > panic: spin lock held too long > cpuid = 5 > time = 1518911834 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfe00f10094d0 > vpanic() at vpanic+0x18d/frame 0xfe00f1009530 > panic() at panic+0x43/frame 0xfe00f1009590 > _mtx_lock_indefinite_check() at _mtx_lock_indefinite_check+0x71/frame > 0xfe00f10095a0 > thread_lock_flags_() at thread_lock_flags_+0xdb/frame 0xfe00f1009610 > statclock_cnt() at statclock_cnt+0xdc/frame 0xfe00f1009650 > handleevents() at handleevents+0x113/frame 0xfe00f10096a0 > timercb() at timercb+0xa9/frame 0xfe00f10096f0 > lapic_handle_timer() at lapic_handle_timer+0xa7/frame 0xfe00f1009730 > timerint_u() at timerint_u+0x96/frame 0xfe00f1009810 > thread_lock_flags_() at thread_lock_flags_+0xc1/frame 0xfe00f1009880 > fork1() at fork1+0x1b9f/frame 0xfe00f1009930 > sys_vfork() at sys_vfork+0x4c/frame 0xfe00f1009980 > amd64_syscall() at amd64_syscall+0xa48/frame 0xfe00f1009ab0 > fast_syscall_common() at fast_syscall_common+0x101/frame 0x7fffcfc0 > KDB: enter: panic > > __curthread () at ./machine/pcpu.h:230 > 230 __asm("movq %%gs:%1,%0" : "=r" (td) > (kgdb) #0 __curthread () at ./machine/pcpu.h:230 > #1 doadump (textdump=-2122191464) at /usr/src/sys/kern/kern_shutdown.c:347 > #2 0x8040b42c in db_fncall_generic (addr=, > rv=, nargs=, args=) > at /usr/src/sys/ddb/db_command.c:609 > #3 db_fncall (dummy1=, dummy2=, > dummy3=, dummy4=) > at /usr/src/sys/ddb/db_command.c:657 > #4 0x8040af79 in db_command (last_cmdp=, > cmd_table=, dopager=) > at /usr/src/sys/ddb/db_command.c:481 > #5 0x8040acf4 in db_command_loop () > at /usr/src/sys/ddb/db_command.c:534 > #6 0x8040df9f in db_trap (type=, code=) > at /usr/src/sys/ddb/db_main.c:250 > #7 0x80b370e3 in kdb_trap (type=3, code=-61456, tf=) > at /usr/src/sys/kern/subr_kdb.c:697 > #8 0x80fa2c5c in trap (frame=0xfe00f1009400) > at /usr/src/sys/amd64/amd64/trap.c:547 > #9 > #10 kdb_enter (why=0x811f280b "panic", msg=) > at /usr/src/sys/kern/subr_kdb.c:479 > #11 0x80aef17a in vpanic (fmt=, ap=0xfe00f1009570) > at /usr/src/sys/kern/kern_shutdown.c:801 > #12 0x80aeefc3 in panic (fmt=0x0) > at /usr/src/sys/kern/kern_shutdown.c:739 > #13 0x80acfa31 in _mtx_lock_indefinite_check (m=, > ldap=) at /usr/src/sys/kern/kern_mutex.c:1224 > #14 0x80acfb9b in thread_lock_flags_ (td=0xf818719d1000, > opts=, file=, line=) > at /usr/src/sys/kern/kern_mutex.c:913 > #15 0x80a89d6c in statclock_cnt (cnt=1, usermode=) > at /usr/src/sys/kern/kern_clock.c:768 > #16 0x810d0003 in handleevents (now=43230207690178, fake=0) > at /usr/src/sys/kern/kern_clocksource.c:196 > #17 0x810d0709 in timercb (et=0x81c528e8 , > arg=) at /usr/src/sys/kern/kern_clocksource.c:353 > #18 0x8110dad7 in lapic_handle_timer (frame=0xfe00f1009740) > a
amd64 head -r329465 (non-debug build, but with symbols): "panic: spin lock held too long" during make check-old, reported during a sys_vfork
This is for FreeBSD running under Hyper-V on a Windows 10 Pro machine. The FreeBSD "disk" bindings are to SSDs, not the insides of NTFS files. 29 logical processors assigned to FreeBSD (on a 32-thread Ryzen Threadripper 1950X). No other Hyper-V use. This happened during: # ~/sys_build_scripts.amd64-host/make_powerpc64vtsc_nodebug_clang_altbinutils-amd64-host.sh check-old DESTDIR=/usr/obj/DESTDIRs/clang-powerpc64-installworld_altbinutils Script started, output file is /root/sys_typescripts/typescript_make_powerpc64vtsc_nodebug_clang_altbinutils-amd64-host-2018-02-17:15:56:20 >>> Checking for old files (Hand typed from a picture of a window's content at slighly different times, expect typos:) KDB: enter: panic [thread pid 42170 tid 100752 ] Stopped at kdb_enter+0x3b: movq $0,kdb_why db> call doadump Dumping 4825 out of 110559 MB: ... (omitted) ... Dump complete = 0 (The "pid 42170" identification as the process reporting the issue does not seem to appear in the core.txt.0 file.) # ls -lTdt /var/crash/* -rw-r--r-- 1 root wheel 100792 Feb 17 16:09:18 2018 /var/crash/core.txt.0 lrwxr-xr-x 1 root wheel 8 Feb 17 16:09:08 2018 /var/crash/vmcore.last -> vmcore.0 lrwxr-xr-x 1 root wheel 6 Feb 17 16:09:08 2018 /var/crash/info.last -> info.0 -rw--- 1 root wheel 5060296704 Feb 17 16:09:08 2018 /var/crash/vmcore.0 -rw--- 1 root wheel 392 Feb 17 16:08:59 2018 /var/crash/info.0 -rw-r--r-- 1 root wheel 2 Feb 17 16:08:59 2018 /var/crash/bounds -rw-r--r-- 1 root wheel 5 Nov 22 04:34:36 2017 /var/crash/minfree >From /var/crash/core.txt.0 : Unread portion of the kernel message buffer: spin lock 0x81b2dcc0 (sched lock 5) held by 0xf8011d936560 (tid 100691) too long panic: spin lock held too long cpuid = 5 time = 1518911834 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfe00f10094d0 vpanic() at vpanic+0x18d/frame 0xfe00f1009530 panic() at panic+0x43/frame 0xfe00f1009590 _mtx_lock_indefinite_check() at _mtx_lock_indefinite_check+0x71/frame 0xfe00f10095a0 thread_lock_flags_() at thread_lock_flags_+0xdb/frame 0xfe00f1009610 statclock_cnt() at statclock_cnt+0xdc/frame 0xfe00f1009650 handleevents() at handleevents+0x113/frame 0xfe00f10096a0 timercb() at timercb+0xa9/frame 0xfe00f10096f0 lapic_handle_timer() at lapic_handle_timer+0xa7/frame 0xfe00f1009730 timerint_u() at timerint_u+0x96/frame 0xfe00f1009810 thread_lock_flags_() at thread_lock_flags_+0xc1/frame 0xfe00f1009880 fork1() at fork1+0x1b9f/frame 0xfe00f1009930 sys_vfork() at sys_vfork+0x4c/frame 0xfe00f1009980 amd64_syscall() at amd64_syscall+0xa48/frame 0xfe00f1009ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0x7fffcfc0 KDB: enter: panic __curthread () at ./machine/pcpu.h:230 230 __asm("movq %%gs:%1,%0" : "=r" (td) (kgdb) #0 __curthread () at ./machine/pcpu.h:230 #1 doadump (textdump=-2122191464) at /usr/src/sys/kern/kern_shutdown.c:347 #2 0x8040b42c in db_fncall_generic (addr=, rv=, nargs=, args=) at /usr/src/sys/ddb/db_command.c:609 #3 db_fncall (dummy1=, dummy2=, dummy3=, dummy4=) at /usr/src/sys/ddb/db_command.c:657 #4 0x8040af79 in db_command (last_cmdp=, cmd_table=, dopager=) at /usr/src/sys/ddb/db_command.c:481 #5 0x8040acf4 in db_command_loop () at /usr/src/sys/ddb/db_command.c:534 #6 0x8040df9f in db_trap (type=, code=) at /usr/src/sys/ddb/db_main.c:250 #7 0x80b370e3 in kdb_trap (type=3, code=-61456, tf=) at /usr/src/sys/kern/subr_kdb.c:697 #8 0x80fa2c5c in trap (frame=0xfe00f1009400) at /usr/src/sys/amd64/amd64/trap.c:547 #9 #10 kdb_enter (why=0x811f280b "panic", msg=) at /usr/src/sys/kern/subr_kdb.c:479 #11 0x80aef17a in vpanic (fmt=, ap=0xfe00f1009570) at /usr/src/sys/kern/kern_shutdown.c:801 #12 0x80aeefc3 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:739 #13 0x80acfa31 in _mtx_lock_indefinite_check (m=, ldap=) at /usr/src/sys/kern/kern_mutex.c:1224 #14 0x80acfb9b in thread_lock_flags_ (td=0xf818719d1000, opts=, file=, line=) at /usr/src/sys/kern/kern_mutex.c:913 #15 0x80a89d6c in statclock_cnt (cnt=1, usermode=) at /usr/src/sys/kern/kern_clock.c:768 #16 0x810d0003 in handleevents (now=43230207690178, fake=0) at /usr/src/sys/kern/kern_clocksource.c:196 #17 0x810d0709 in timercb (et=0x81c528e8 , arg=) at /usr/src/sys/kern/kern_clocksource.c:353 #18 0x8110dad7 in lapic_handle_timer (frame=0xfe00f1009740) at /usr/src/sys/x86/x86/local_apic.c:1305 #19 0x80f849d0 in timerint_u () at /usr/src/sys/amd64/amd64/apic_vector.S:132 #20 0xfe00f1009828 in ?? () #21 0xb6b1 in ?? () #22 0x2000 in ?? () #23 0xdfff in ?? () #24 0x00c11c08e43e7fd5 in ?? () #25 0x03e