Re: debug head -r368500 kernel (for example) : "lock order reversal: (sleepable after non-sleepable)" involving ure0 and a sysctl lock; more
On 2020-Dec-19, at 03:09, Hans Petter Selasky wrote: > Please test: > > https://svnweb.freebsd.org/changeset/base/368799 > https://svnweb.freebsd.org/changeset/base/368801 > > --HPS I grabbed a debug -r368803 kernel from artifacts (first arm64 snapshot available containing the above 2 updates): https://artifact.ci.freebsd.org/snapshot/13.0-CURRENT/r368803/arm64/aarch64/kernel.txz I used it to substitute in the updated debug kernel and booted the same configuration. It booted fine with no LOR or uma_zalloc_debug related console output. I've not tried my own non-debug build yet but that kind of context had never given me a clue of these issues anyway. (I am not expecting the above to change the non-debug "Root mount waiting for: usbus0" for the uefi/ACPI USB3 SSD based boot sequence.) === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: debug head -r368500 kernel (for example) : "lock order reversal: (sleepable after non-sleepable)" involving ure0 and a sysctl lock; more
Please test: https://svnweb.freebsd.org/changeset/base/368799 https://svnweb.freebsd.org/changeset/base/368801 --HPS ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: debug head -r368500 kernel (for example) : "lock order reversal: (sleepable after non-sleepable)" involving ure0 and a sysctl lock; more
[I managed to not cc the primary person that I intended but to cc the secondary person. So this resend just adds jmg and removes hps.] On 2020-Dec-18, at 22:27, Mark Millard wrote: > The following is from head -r368500's artifact kernel from: > > https://artifact.ci.freebsd.org/snapshot/13.0-CURRENT/r368500/arm64/aarch64/kernel.txz > > but the same sort of material showed for -r368000 . > (I was attempting a bisect for a different issue but > the debug kernels did not fail for what I was looking > for and all the debug versions that I tried reported > similarly to the below.) > > Note also the mixing in of "uma_zalloc_debug" activity > after the initial LOR backtrace, ure0 still involved. > > Autoloading module: if_ure.ko > ure0 on uhub0 > ure0: on > usbus0 > add host 127.0.0.1: gatelock order reversal: (sleepable after non-sleepable) > 1st 0xa2b2cea0 ure0 (ure0, sleep mutex) @ > /usr/src/sys/dev/usb/usb_request.c:714 > 2nd 0x00dd6858 sysctl lock (sysctl lock, sleepable rm) @ /usway lo0 > fib 0: route alrr/src/sys/kern/kern_sysctl.c:836 > lock order ure0 -> sysctl lock attempted at: > #0 0x0056d068 at witness_checkorder+0xc54 > #1 0x004f8f08 at _rm_wlock_debug+0x88 > #2 0x0050ee2c at sysctl_add_oid+0x60 > #3 0x9e415780 at ure_attach_post+0x1a78 > #4 0x00391d6c at ue_attach_post_task+0x3c > #5 0x003826e0 at usb_process+0x10c > #6 0x004baa1c at fork_exit+0x7c > #7 0x00816544 at fork_trampoline+0x10 > uma_zalloc_debug: zone "malloc-128eady in table > " with the following non-sleepable locks held: > exclusive sleep mutex ure0 (ure0) r = 0 (0xa2b2cea0) locked @ > /usr/src/sys/dev/usb/usb_request.c:714 > stack backtrace: > #0 0x0056d388 at witness_debugger+0x64 > #1 0x0056e518 at witness_warn+0x3ec > #2 0x00778f9c at uma_zalloc_debug+0x2c > #3 0x00778998 at uma_zalloc_arg+0x2c > #4 0x004d534c at malloc+0xa0 > #5 0x0050ee80 at sysctl_add_oid+0xb4 > #6 0x9e415780 at ure_attach_post+0x1a78 > #7 0x00391d6c at ue_attach_post_task+0x3c > #8 0x003826e0 at usb_process+0x10c > #9 0x004baa1c at fork_exit+0x7c > #10 0x00816544 at fork_trampoline+0x10 > uma_zalloc_debug: zone "malloc-16" with the following non-sleepable locks > held: > exclusive sleep mutex ure0 (ure0) r = 0 (0xa2b2cea0) locked @ > /usr/src/sys/dev/usb/usb_request.c:714 > stack backtrace: > #0 0x0056d388 at witness_debugger+0x64 > #1 0x0056e518 at witness_warn+0x3ec > #2 0x00778f9c at uma_zalloc_debug+0x2c > #3 0x00778998 at uma_zalloc_arg+0x2c > #4 0x004d534c at malloc+0xa0 > #5 0x005f8c80 at strdup+0x2c > #6 0x0050eeb8 at sysctl_add_oid+0xec > #7 0x9e415780 at ure_attach_post+0x1a78 > #8 0x00391d6c at ue_attach_post_task+0x3c > #9 0x003826e0 at usb_process+0x10c > #10 0x004baa1c at fork_exit+0x7c > #11 0x00816544 at fork_trampoline+0x10 > uma_zalloc_debug: zone "malloc-64" with the following non-sleepable locks > held: > exclusive sleep mutex ure0 (ure0) r = 0 (0xa2b2cea0) locked @ > /usr/src/sys/dev/usb/usb_request.c:714 > stack backtrace: > #0 0x0056d388 at witness_debugger+0x64 > #1 0x0056e518 at witness_warn+0x3ec > #2 0x00778f9c at uma_zalloc_debug+0x2c > #3 0x00778998 at uma_zalloc_arg+0x2c > #4 0x004d534c at malloc+0xa0 > #5 0x005f8c80 at strdup+0x2c > #6 0x0050eee4 at sysctl_add_oid+0x118 > #7 0x9e415780 at ure_attach_post+0x1a78 > #8 0x00391d6c at ue_attach_post_task+0x3c > #9 0x003826e0 at usb_process+0x10c > #10 0x004baa1c at fork_exit+0x7c > #11 0x00816544 at fork_trampoline+0x10 > uma_zalloc_debug: zone "malloc-32" with the following non-sleepable locks > held: > exclusive sleep mutex ure0 (ure0) r = 0 (0xa2b2cea0) locked @ > /usr/src/sys/dev/usb/usb_request.c:714 > stack backtrace: > #0 0x0056d388 at witness_debugger+0x64 > #1 0x0056e518 at witness_warn+0x3ec > #2 0x00778f9c at uma_zalloc_debug+0x2c > #3 0x00778998 at uma_zalloc_arg+0x2c > #4 0x004d534c at malloc+0xa0 > #5 0x0050ef3c at sysctl_add_oid+0x170 > #6 0x9e415780 at ure_attach_post+0x1a78 > #7 0x00391d6c at ue_attach_post_task+0x3c > #8 0x003826e0 at usb_process+0x10c > #9 0x004baa1c at fork_exit+0x7c > #10 0x00816544 at fork_trampoline+0x10 > miibus0: on ure0 > rgephy0: PHY 0 on miibus0 > add hrgephy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, > 1000baseT-FDX, 1000baseT-FDX-master, auto > 0 fib 0: route already iue0: on ure0 > > > > The context here is an RPi4 (aarch64 cortex-a72) with: > > # uname -apKU > FreeBSD RPi4B 13.0-CURRENT FreeBSD 13.0-CURRENT #0 r368500: Thu Dec 10 > 07:52:39 UTC 2020 >
debug head -r368500 kernel (for example) : "lock order reversal: (sleepable after non-sleepable)" involving ure0 and a sysctl lock; more
The following is from head -r368500's artifact kernel from: https://artifact.ci.freebsd.org/snapshot/13.0-CURRENT/r368500/arm64/aarch64/kernel.txz but the same sort of material showed for -r368000 . (I was attempting a bisect for a different issue but the debug kernels did not fail for what I was looking for and all the debug versions that I tried reported similarly to the below.) Note also the mixing in of "uma_zalloc_debug" activity after the initial LOR backtrace, ure0 still involved. Autoloading module: if_ure.ko ure0 on uhub0 ure0: on usbus0 add host 127.0.0.1: gatelock order reversal: (sleepable after non-sleepable) 1st 0xa2b2cea0 ure0 (ure0, sleep mutex) @ /usr/src/sys/dev/usb/usb_request.c:714 2nd 0x00dd6858 sysctl lock (sysctl lock, sleepable rm) @ /usway lo0 fib 0: route alrr/src/sys/kern/kern_sysctl.c:836 lock order ure0 -> sysctl lock attempted at: #0 0x0056d068 at witness_checkorder+0xc54 #1 0x004f8f08 at _rm_wlock_debug+0x88 #2 0x0050ee2c at sysctl_add_oid+0x60 #3 0x9e415780 at ure_attach_post+0x1a78 #4 0x00391d6c at ue_attach_post_task+0x3c #5 0x003826e0 at usb_process+0x10c #6 0x004baa1c at fork_exit+0x7c #7 0x00816544 at fork_trampoline+0x10 uma_zalloc_debug: zone "malloc-128eady in table " with the following non-sleepable locks held: exclusive sleep mutex ure0 (ure0) r = 0 (0xa2b2cea0) locked @ /usr/src/sys/dev/usb/usb_request.c:714 stack backtrace: #0 0x0056d388 at witness_debugger+0x64 #1 0x0056e518 at witness_warn+0x3ec #2 0x00778f9c at uma_zalloc_debug+0x2c #3 0x00778998 at uma_zalloc_arg+0x2c #4 0x004d534c at malloc+0xa0 #5 0x0050ee80 at sysctl_add_oid+0xb4 #6 0x9e415780 at ure_attach_post+0x1a78 #7 0x00391d6c at ue_attach_post_task+0x3c #8 0x003826e0 at usb_process+0x10c #9 0x004baa1c at fork_exit+0x7c #10 0x00816544 at fork_trampoline+0x10 uma_zalloc_debug: zone "malloc-16" with the following non-sleepable locks held: exclusive sleep mutex ure0 (ure0) r = 0 (0xa2b2cea0) locked @ /usr/src/sys/dev/usb/usb_request.c:714 stack backtrace: #0 0x0056d388 at witness_debugger+0x64 #1 0x0056e518 at witness_warn+0x3ec #2 0x00778f9c at uma_zalloc_debug+0x2c #3 0x00778998 at uma_zalloc_arg+0x2c #4 0x004d534c at malloc+0xa0 #5 0x005f8c80 at strdup+0x2c #6 0x0050eeb8 at sysctl_add_oid+0xec #7 0x9e415780 at ure_attach_post+0x1a78 #8 0x00391d6c at ue_attach_post_task+0x3c #9 0x003826e0 at usb_process+0x10c #10 0x004baa1c at fork_exit+0x7c #11 0x00816544 at fork_trampoline+0x10 uma_zalloc_debug: zone "malloc-64" with the following non-sleepable locks held: exclusive sleep mutex ure0 (ure0) r = 0 (0xa2b2cea0) locked @ /usr/src/sys/dev/usb/usb_request.c:714 stack backtrace: #0 0x0056d388 at witness_debugger+0x64 #1 0x0056e518 at witness_warn+0x3ec #2 0x00778f9c at uma_zalloc_debug+0x2c #3 0x00778998 at uma_zalloc_arg+0x2c #4 0x004d534c at malloc+0xa0 #5 0x005f8c80 at strdup+0x2c #6 0x0050eee4 at sysctl_add_oid+0x118 #7 0x9e415780 at ure_attach_post+0x1a78 #8 0x00391d6c at ue_attach_post_task+0x3c #9 0x003826e0 at usb_process+0x10c #10 0x004baa1c at fork_exit+0x7c #11 0x00816544 at fork_trampoline+0x10 uma_zalloc_debug: zone "malloc-32" with the following non-sleepable locks held: exclusive sleep mutex ure0 (ure0) r = 0 (0xa2b2cea0) locked @ /usr/src/sys/dev/usb/usb_request.c:714 stack backtrace: #0 0x0056d388 at witness_debugger+0x64 #1 0x0056e518 at witness_warn+0x3ec #2 0x00778f9c at uma_zalloc_debug+0x2c #3 0x00778998 at uma_zalloc_arg+0x2c #4 0x004d534c at malloc+0xa0 #5 0x0050ef3c at sysctl_add_oid+0x170 #6 0x9e415780 at ure_attach_post+0x1a78 #7 0x00391d6c at ue_attach_post_task+0x3c #8 0x003826e0 at usb_process+0x10c #9 0x004baa1c at fork_exit+0x7c #10 0x00816544 at fork_trampoline+0x10 miibus0: on ure0 rgephy0: PHY 0 on miibus0 add hrgephy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto 0 fib 0: route already iue0: on ure0 The context here is an RPi4 (aarch64 cortex-a72) with: # uname -apKU FreeBSD RPi4B 13.0-CURRENT FreeBSD 13.0-CURRENT #0 r368500: Thu Dec 10 07:52:39 UTC 2020 r...@freebsd-head-aarch64-build.jail.ci.freebsd.org:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC arm64 aarch64 1300131 1300131 The boot attempts were via uefi/ACPI using https://github.com/pftf/RPi4 v1.21 materials, directly booting from the USB3 SSD, no microsd card involved. Some context in case it contributes something for the above (probably not) . . . The reason for the bisect was: such boot attempts fail to mount route with my non-debug head -r368500 kernel
