Re: panic: Unaligned free (was: kernel panic while copying files)
On Wed, 7 Jul 2021 09:38:05 +0100 Edward Tomasz Napiera?a wrote: > On 0705T1833, Gary Jennejohn wrote: > > On Mon, 5 Jul 2021 15:04:48 +0100 > > Edward Tomasz Napiera__a wrote: > > > > > On 0701T1330, Gary Jennejohn wrote: > > > > Gary Jennejohn wrote: > > > > > I noticed that the value of vm.debug.divisor affects what value is > > > > > returned in uma_core.c:uma_dbg_kskip(), so I decided to try a few > > > > > different values. > > > > > > > > > > The returned value is used to set skipdbg in uma_core.c:item_dtor(). > > > > > > > > > > The default is vm.debug.divisor=1. > > > > > > > > > > vm.debug.divisor is only present when INVARIANTS is defined. > > > > > > > > > > kskipdbg eventually affects the value of freei. > > > > > > > > > > With these values: > > > > > vm.debug.divisor: 0 > > > > > kern.cam.da.enable_uma_ccbs: 1 > > > > > I can turn on the disk and it comes up without a panic! > > > > > > > > > > However, I didn't try to do any large data transfers to the disk. > > > > > > > > > > So, it appears that at least vm.debug.divisor is a big factor in > > > > > whether or not a panic happens with INVARIANTS. > > > > > > > > > > > > > I decided to do a real test. So I built a kernel w/o INVARIANTS and > > > > installed it to /boot/test. > > > > > > > > Then I stuck a 160GB disk I had around into an external USB3 enclosure > > > > and put a filesystem on it. > > > > > > > > The I booted the new kernel from /boot/test and set the sysctls so: > > > > kern.cam.da.enable_uma_ccbs: 1 > > > > kern.cam.ada.enable_uma_ccbs: 1 > > > > > > > > After that I plugged in the external USB3 enclosure and copied about > > > > 114GiB of data from an internal SSD to it - without a kernel panic: > > > > FilesystemSizeUsed Avail Capacity Mounted on > > > > /dev/da0p1144G114G 18G86%/mnt > > > > > > > > I'm pretty sure that's more than I could copy without a kernel panic > > > > prior to the recent changes made in cam and umass. > > > > > > > > My test may not be real proof that all bugs have been squashed, but it > > > > certainly seems to be a better situation than we had before. > > > > > > I think the vm.debug.divisor simply masks the problem; the underlying > > > bug is still there. > > > > > > Could you go back to the setup which panics, and then test the patch > > > at https://reviews.freebsd.org/D31054? It fixes the scenario described > > > by Warner. > > > > > > > It looks like this patch fixes things. > > > > I used the default value vm.debug.divisor=1 and both enable_uma_ccbs=1 > > (which are now the default values on my system). > > > > I used the 8TiB disk, which spins up very slowly and usually resulted very > > quickly in a panic - no panic with the patch. > > > > Then using dd to /dev/null (bs=1m) I transferred: > > > > 308755+0 records in > > 308755+0 records out > > 323753082880 bytes transferred in 1366.162410 secs (236979938 bytes/sec) > > > > from the disk, so about 324GiB without a panic. > > Perfect, I've committed the fix. Thank you! > Thanks to you! I built a new kernel as soon as I saw the commit and am running it since yesterday. -- Gary Jennejohn
Re: panic: Unaligned free (was: kernel panic while copying files)
On 0705T1833, Gary Jennejohn wrote: > On Mon, 5 Jul 2021 15:04:48 +0100 > Edward Tomasz Napiera__a wrote: > > > On 0701T1330, Gary Jennejohn wrote: > > > Gary Jennejohn wrote: > > > > I noticed that the value of vm.debug.divisor affects what value is > > > > returned in uma_core.c:uma_dbg_kskip(), so I decided to try a few > > > > different values. > > > > > > > > The returned value is used to set skipdbg in uma_core.c:item_dtor(). > > > > > > > > The default is vm.debug.divisor=1. > > > > > > > > vm.debug.divisor is only present when INVARIANTS is defined. > > > > > > > > kskipdbg eventually affects the value of freei. > > > > > > > > With these values: > > > > vm.debug.divisor: 0 > > > > kern.cam.da.enable_uma_ccbs: 1 > > > > I can turn on the disk and it comes up without a panic! > > > > > > > > However, I didn't try to do any large data transfers to the disk. > > > > > > > > So, it appears that at least vm.debug.divisor is a big factor in > > > > whether or not a panic happens with INVARIANTS. > > > > > > > > > > I decided to do a real test. So I built a kernel w/o INVARIANTS and > > > installed it to /boot/test. > > > > > > Then I stuck a 160GB disk I had around into an external USB3 enclosure > > > and put a filesystem on it. > > > > > > The I booted the new kernel from /boot/test and set the sysctls so: > > > kern.cam.da.enable_uma_ccbs: 1 > > > kern.cam.ada.enable_uma_ccbs: 1 > > > > > > After that I plugged in the external USB3 enclosure and copied about > > > 114GiB of data from an internal SSD to it - without a kernel panic: > > > FilesystemSizeUsed Avail Capacity Mounted on > > > /dev/da0p1144G114G 18G86%/mnt > > > > > > I'm pretty sure that's more than I could copy without a kernel panic > > > prior to the recent changes made in cam and umass. > > > > > > My test may not be real proof that all bugs have been squashed, but it > > > certainly seems to be a better situation than we had before. > > > > I think the vm.debug.divisor simply masks the problem; the underlying > > bug is still there. > > > > Could you go back to the setup which panics, and then test the patch > > at https://reviews.freebsd.org/D31054? It fixes the scenario described > > by Warner. > > > > It looks like this patch fixes things. > > I used the default value vm.debug.divisor=1 and both enable_uma_ccbs=1 > (which are now the default values on my system). > > I used the 8TiB disk, which spins up very slowly and usually resulted very > quickly in a panic - no panic with the patch. > > Then using dd to /dev/null (bs=1m) I transferred: > > 308755+0 records in > 308755+0 records out > 323753082880 bytes transferred in 1366.162410 secs (236979938 bytes/sec) > > from the disk, so about 324GiB without a panic. Perfect, I've committed the fix. Thank you!
Re: panic: Unaligned free (was: kernel panic while copying files)
On Mon, 5 Jul 2021 15:04:48 +0100 Edward Tomasz Napiera__a wrote: > On 0701T1330, Gary Jennejohn wrote: > > Gary Jennejohn wrote: > > > I noticed that the value of vm.debug.divisor affects what value is > > > returned in uma_core.c:uma_dbg_kskip(), so I decided to try a few > > > different values. > > > > > > The returned value is used to set skipdbg in uma_core.c:item_dtor(). > > > > > > The default is vm.debug.divisor=1. > > > > > > vm.debug.divisor is only present when INVARIANTS is defined. > > > > > > kskipdbg eventually affects the value of freei. > > > > > > With these values: > > > vm.debug.divisor: 0 > > > kern.cam.da.enable_uma_ccbs: 1 > > > I can turn on the disk and it comes up without a panic! > > > > > > However, I didn't try to do any large data transfers to the disk. > > > > > > So, it appears that at least vm.debug.divisor is a big factor in > > > whether or not a panic happens with INVARIANTS. > > > > > > > I decided to do a real test. So I built a kernel w/o INVARIANTS and > > installed it to /boot/test. > > > > Then I stuck a 160GB disk I had around into an external USB3 enclosure > > and put a filesystem on it. > > > > The I booted the new kernel from /boot/test and set the sysctls so: > > kern.cam.da.enable_uma_ccbs: 1 > > kern.cam.ada.enable_uma_ccbs: 1 > > > > After that I plugged in the external USB3 enclosure and copied about > > 114GiB of data from an internal SSD to it - without a kernel panic: > > FilesystemSizeUsed Avail Capacity Mounted on > > /dev/da0p1144G114G 18G86%/mnt > > > > I'm pretty sure that's more than I could copy without a kernel panic > > prior to the recent changes made in cam and umass. > > > > My test may not be real proof that all bugs have been squashed, but it > > certainly seems to be a better situation than we had before. > > I think the vm.debug.divisor simply masks the problem; the underlying > bug is still there. > > Could you go back to the setup which panics, and then test the patch > at https://reviews.freebsd.org/D31054? It fixes the scenario described > by Warner. > It looks like this patch fixes things. I used the default value vm.debug.divisor=1 and both enable_uma_ccbs=1 (which are now the default values on my system). I used the 8TiB disk, which spins up very slowly and usually resulted very quickly in a panic - no panic with the patch. Then using dd to /dev/null (bs=1m) I transferred: 308755+0 records in 308755+0 records out 323753082880 bytes transferred in 1366.162410 secs (236979938 bytes/sec) from the disk, so about 324GiB without a panic. -- Gary Jennejohn
Re: panic: Unaligned free (was: kernel panic while copying files)
On 0701T1330, Gary Jennejohn wrote: > Gary Jennejohn wrote: > > I noticed that the value of vm.debug.divisor affects what value is > > returned in uma_core.c:uma_dbg_kskip(), so I decided to try a few > > different values. > > > > The returned value is used to set skipdbg in uma_core.c:item_dtor(). > > > > The default is vm.debug.divisor=1. > > > > vm.debug.divisor is only present when INVARIANTS is defined. > > > > kskipdbg eventually affects the value of freei. > > > > With these values: > > vm.debug.divisor: 0 > > kern.cam.da.enable_uma_ccbs: 1 > > I can turn on the disk and it comes up without a panic! > > > > However, I didn't try to do any large data transfers to the disk. > > > > So, it appears that at least vm.debug.divisor is a big factor in > > whether or not a panic happens with INVARIANTS. > > > > I decided to do a real test. So I built a kernel w/o INVARIANTS and > installed it to /boot/test. > > Then I stuck a 160GB disk I had around into an external USB3 enclosure > and put a filesystem on it. > > The I booted the new kernel from /boot/test and set the sysctls so: > kern.cam.da.enable_uma_ccbs: 1 > kern.cam.ada.enable_uma_ccbs: 1 > > After that I plugged in the external USB3 enclosure and copied about > 114GiB of data from an internal SSD to it - without a kernel panic: > FilesystemSizeUsed Avail Capacity Mounted on > /dev/da0p1144G114G 18G86%/mnt > > I'm pretty sure that's more than I could copy without a kernel panic > prior to the recent changes made in cam and umass. > > My test may not be real proof that all bugs have been squashed, but it > certainly seems to be a better situation than we had before. I think the vm.debug.divisor simply masks the problem; the underlying bug is still there. Could you go back to the setup which panics, and then test the patch at https://reviews.freebsd.org/D31054? It fixes the scenario described by Warner.
Re: panic: Unaligned free (was: kernel panic while copying files)
Gary Jennejohn wrote: > On Wed, 30 Jun 2021 10:35:14 -0600 > Warner Losh wrote: > > > On Wed, Jun 30, 2021 at 6:58 AM Gary Jennejohn wrote: > > > > > On Wed, 30 Jun 2021 06:02:59 +0100 > > > Graham Perrin wrote: > > > > > > > On 29/06/2021 10:42, Gary Jennejohn wrote: > > > > > ___ panic is now the result of an unaligned free. > > > > > > > > > > panic: Unaligned free of 0xf800259e2800 from zone > > > > > 0xfe00dc9d2000(da_ccb) slab 0xf800259e2fd8(3) > > > > > > > > > > I have the crash dump and a debug kernel in case anyone wants more > > > info. > > > > Can you post the backtrace etc. here? Thanks > > > > > > > > > > Sure. As can be seen from the uma zone being da_ccb, the panic > > > resulted from setting kern.cam.da.enable_uma_ccbs=1. > > > > > > Unread portion of the kernel message buffer: > > > panic: Unaligned free of 0xf800259e2800 from zone > > > 0xfe00dc9d2000(da_ccb) slab 0xf800259e2fd8(3) > > > cpuid = 2 > > > time = 1624958650 > > > KDB: stack backtrace: > > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2c/frame > > > 0xfe00c62687a0 > > > kdb_backtrace() at kdb_backtrace+0x46/frame 0xfe00c6268850 > > > vpanic() at vpanic+0x227/frame 0xfe00c62688f0 > > > panic() at panic+0x4e/frame 0xfe00c6268950 > > > uma_dbg_free() at uma_dbg_free+0xfc/frame 0xfe00c62689a0 > > > item_dtor() at item_dtor+0x7c/frame 0xfe00c62689e0 > > > uma_zfree_arg() at uma_zfree_arg+0xf0/frame 0xfe00c6268a50 > > > uma_zfree() at uma_zfree+0x23/frame 0xfe00c6268a70 > > > xpt_free_ccb() at xpt_free_ccb+0x43/frame 0xfe00c6268a90 > > > camperiphdone() at camperiphdone+0x211/frame 0xfe00c6268ae0 > > > xpt_done_process() at xpt_done_process+0x550/frame 0xfe00c6268b40 > > > xpt_done_td() at xpt_done_td+0x1c0/frame 0xfe00c6268b80 > > > fork_exit() at fork_exit+0x117/frame 0xfe00c6268bf0 > > > fork_trampoline() at fork_trampoline+0xe/frame 0xfe00c6268bf0 > > > --- trap 0, rip = 0, rsp = 0, rbp = 0 --- > > > KDB: enter: panic > > > > > > doadump (textdump=0) at /usr/src/sys/kern/kern_shutdown.c:399 > > > 399 dumptid = curthread->td_tid; > > > (kgdb) bt > > > #0 doadump (textdump=0) at /usr/src/sys/kern/kern_shutdown.c:399 > > > #1 0x804d5dd7 in db_dump (dummy=-2138843371, dummy2=false, > > > dummy3=-1, > > > dummy4=0xfe00c6268320 "") at /usr/src/sys/ddb/db_command.c:575 > > > #2 0x804d5bf4 in db_command ( > > > last_cmdp=0x8114ce80 , cmd_table=0x0, > > > dopager=1) > > > at /usr/src/sys/ddb/db_command.c:482 > > > #3 0x804d583c in db_command_loop () > > > at /usr/src/sys/ddb/db_command.c:535 > > > #4 0x804da27c in db_trap (type=3, code=0) > > > at /usr/src/sys/ddb/db_main.c:270 > > > #5 0x8083df9d in kdb_trap (type=3, code=0, tf=0xfe00c6268770) > > > at /usr/src/sys/kern/subr_kdb.c:727 > > > #6 0x80d31494 in trap (frame=0xfe00c6268770) > > > at /usr/src/sys/amd64/amd64/trap.c:604 > > > #7 0x80d32628 in trap_check (frame=0xfe00c6268770) > > > at /usr/src/sys/amd64/amd64/trap.c:664 > > > #8 > > > #9 breakpoint () at /usr/src/sys/amd64/include/cpufunc.h:66 > > > #10 0x8083d3d0 in kdb_enter (why=0x80e0355b "panic", > > > msg=0x80e0355b "panic") at /usr/src/sys/kern/subr_kdb.c:505 > > > #11 0x807d1725 in vpanic ( > > > fmt=0x80dbca46 "Unaligned free of %p from zone %p(%s) slab > > > %p(%d)", ap=0xfe00c6268930) at /usr/src/sys/kern/kern_shutdown.c:906 > > > #12 0x807d120e in panic ( > > > fmt=0x80dbca46 "Unaligned free of %p from zone %p(%s) slab > > > %p(%d)") > > > at /usr/src/sys/kern/kern_shutdown.c:843 > > > #13 0x80c16a8c in uma_dbg_free (zone=0xfe00dc9d2000, > > > slab=0xf800259e2fd8, item=0xf800259e2800) > > > at /usr/src/sys/vm/uma_core.c:5659 > > > #14 0x80c0c5dc in item_dtor (zone=0xfe00dc9d2000, > > > item=0xf800259e2800, size=544, udata=0x0, skip=SKIP_NONE) > > > at /usr/src/sys/vm/uma_core.c:3418 > > > #15 0x80c0ba60 in uma_zfree_arg (zone=0xfe00dc9d2000, > > > item=0xf800259e2800, udata=0x0) at /usr/src/sys/vm/uma_core.c:4374 > > > #16 0x802e45d3 in uma_zfree (zone=0xfe00dc9d2000, > > > item=0xf800259e2800) at /usr/src/sys/vm/uma.h:404 > > > #17 0x802dc3c3 in xpt_free_ccb (free_ccb=0xf800259e2800) > > > at /usr/src/sys/cam/cam_xpt.c:4676 > > > #18 0x802dacf1 in camperiphdone (periph=0xf80025329b00, > > > done_ccb=0xf80025a24cc0) at /usr/src/sys/cam/cam_periph.c:1427 > > > #19 0x802e4520 in xpt_done_process (ccb_h=0xf80025a24cc0) > > > at /usr/src/sys/cam/cam_xpt.c:5493 > > > #20 0x802e68e0 in xpt_done_td (arg=0x81143700 > > > ) > > > at /usr/src/sys/cam/cam_xpt.c:5548 > > > #21 0x807673c7 in fork_exit
Re: panic: Unaligned free (was: kernel panic while copying files)
On Wed, 30 Jun 2021 10:35:14 -0600 Warner Losh wrote: > On Wed, Jun 30, 2021 at 6:58 AM Gary Jennejohn wrote: > > > On Wed, 30 Jun 2021 06:02:59 +0100 > > Graham Perrin wrote: > > > > > On 29/06/2021 10:42, Gary Jennejohn wrote: > > > > ___ panic is now the result of an unaligned free. > > > > > > > > panic: Unaligned free of 0xf800259e2800 from zone > > > > 0xfe00dc9d2000(da_ccb) slab 0xf800259e2fd8(3) > > > > > > > > I have the crash dump and a debug kernel in case anyone wants more > > info. > > > Can you post the backtrace etc. here? Thanks > > > > > > > Sure. As can be seen from the uma zone being da_ccb, the panic > > resulted from setting kern.cam.da.enable_uma_ccbs=1. > > > > Unread portion of the kernel message buffer: > > panic: Unaligned free of 0xf800259e2800 from zone > > 0xfe00dc9d2000(da_ccb) slab 0xf800259e2fd8(3) > > cpuid = 2 > > time = 1624958650 > > KDB: stack backtrace: > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2c/frame > > 0xfe00c62687a0 > > kdb_backtrace() at kdb_backtrace+0x46/frame 0xfe00c6268850 > > vpanic() at vpanic+0x227/frame 0xfe00c62688f0 > > panic() at panic+0x4e/frame 0xfe00c6268950 > > uma_dbg_free() at uma_dbg_free+0xfc/frame 0xfe00c62689a0 > > item_dtor() at item_dtor+0x7c/frame 0xfe00c62689e0 > > uma_zfree_arg() at uma_zfree_arg+0xf0/frame 0xfe00c6268a50 > > uma_zfree() at uma_zfree+0x23/frame 0xfe00c6268a70 > > xpt_free_ccb() at xpt_free_ccb+0x43/frame 0xfe00c6268a90 > > camperiphdone() at camperiphdone+0x211/frame 0xfe00c6268ae0 > > xpt_done_process() at xpt_done_process+0x550/frame 0xfe00c6268b40 > > xpt_done_td() at xpt_done_td+0x1c0/frame 0xfe00c6268b80 > > fork_exit() at fork_exit+0x117/frame 0xfe00c6268bf0 > > fork_trampoline() at fork_trampoline+0xe/frame 0xfe00c6268bf0 > > --- trap 0, rip = 0, rsp = 0, rbp = 0 --- > > KDB: enter: panic > > > > doadump (textdump=0) at /usr/src/sys/kern/kern_shutdown.c:399 > > 399 dumptid = curthread->td_tid; > > (kgdb) bt > > #0 doadump (textdump=0) at /usr/src/sys/kern/kern_shutdown.c:399 > > #1 0x804d5dd7 in db_dump (dummy=-2138843371, dummy2=false, > > dummy3=-1, > > dummy4=0xfe00c6268320 "") at /usr/src/sys/ddb/db_command.c:575 > > #2 0x804d5bf4 in db_command ( > > last_cmdp=0x8114ce80 , cmd_table=0x0, > > dopager=1) > > at /usr/src/sys/ddb/db_command.c:482 > > #3 0x804d583c in db_command_loop () > > at /usr/src/sys/ddb/db_command.c:535 > > #4 0x804da27c in db_trap (type=3, code=0) > > at /usr/src/sys/ddb/db_main.c:270 > > #5 0x8083df9d in kdb_trap (type=3, code=0, tf=0xfe00c6268770) > > at /usr/src/sys/kern/subr_kdb.c:727 > > #6 0x80d31494 in trap (frame=0xfe00c6268770) > > at /usr/src/sys/amd64/amd64/trap.c:604 > > #7 0x80d32628 in trap_check (frame=0xfe00c6268770) > > at /usr/src/sys/amd64/amd64/trap.c:664 > > #8 > > #9 breakpoint () at /usr/src/sys/amd64/include/cpufunc.h:66 > > #10 0x8083d3d0 in kdb_enter (why=0x80e0355b "panic", > > msg=0x80e0355b "panic") at /usr/src/sys/kern/subr_kdb.c:505 > > #11 0x807d1725 in vpanic ( > > fmt=0x80dbca46 "Unaligned free of %p from zone %p(%s) slab > > %p(%d)", ap=0xfe00c6268930) at /usr/src/sys/kern/kern_shutdown.c:906 > > #12 0x807d120e in panic ( > > fmt=0x80dbca46 "Unaligned free of %p from zone %p(%s) slab > > %p(%d)") > > at /usr/src/sys/kern/kern_shutdown.c:843 > > #13 0x80c16a8c in uma_dbg_free (zone=0xfe00dc9d2000, > > slab=0xf800259e2fd8, item=0xf800259e2800) > > at /usr/src/sys/vm/uma_core.c:5659 > > #14 0x80c0c5dc in item_dtor (zone=0xfe00dc9d2000, > > item=0xf800259e2800, size=544, udata=0x0, skip=SKIP_NONE) > > at /usr/src/sys/vm/uma_core.c:3418 > > #15 0x80c0ba60 in uma_zfree_arg (zone=0xfe00dc9d2000, > > item=0xf800259e2800, udata=0x0) at /usr/src/sys/vm/uma_core.c:4374 > > #16 0x802e45d3 in uma_zfree (zone=0xfe00dc9d2000, > > item=0xf800259e2800) at /usr/src/sys/vm/uma.h:404 > > #17 0x802dc3c3 in xpt_free_ccb (free_ccb=0xf800259e2800) > > at /usr/src/sys/cam/cam_xpt.c:4676 > > #18 0x802dacf1 in camperiphdone (periph=0xf80025329b00, > > done_ccb=0xf80025a24cc0) at /usr/src/sys/cam/cam_periph.c:1427 > > #19 0x802e4520 in xpt_done_process (ccb_h=0xf80025a24cc0) > > at /usr/src/sys/cam/cam_xpt.c:5493 > > #20 0x802e68e0 in xpt_done_td (arg=0x81143700 ) > > at /usr/src/sys/cam/cam_xpt.c:5548 > > #21 0x807673c7 in fork_exit (callout=0x802e6720 > > , > > arg=0x81143700 , frame=0xfe00c6268c00) > > at /usr/src/sys/kern/kern_fork.c:1083 > > #22 > > > > [kgdb stuff removed] > > > > (kgdb) down > > #15 0x80c0ba60 in uma_zfree_arg
Re: panic: Unaligned free (was: kernel panic while copying files)
On Wed, Jun 30, 2021 at 6:58 AM Gary Jennejohn wrote: > On Wed, 30 Jun 2021 06:02:59 +0100 > Graham Perrin wrote: > > > On 29/06/2021 10:42, Gary Jennejohn wrote: > > > ___ panic is now the result of an unaligned free. > > > > > > panic: Unaligned free of 0xf800259e2800 from zone > > > 0xfe00dc9d2000(da_ccb) slab 0xf800259e2fd8(3) > > > > > > I have the crash dump and a debug kernel in case anyone wants more > info. > > Can you post the backtrace etc. here? Thanks > > > > Sure. As can be seen from the uma zone being da_ccb, the panic > resulted from setting kern.cam.da.enable_uma_ccbs=1. > > Unread portion of the kernel message buffer: > panic: Unaligned free of 0xf800259e2800 from zone > 0xfe00dc9d2000(da_ccb) slab 0xf800259e2fd8(3) > cpuid = 2 > time = 1624958650 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2c/frame > 0xfe00c62687a0 > kdb_backtrace() at kdb_backtrace+0x46/frame 0xfe00c6268850 > vpanic() at vpanic+0x227/frame 0xfe00c62688f0 > panic() at panic+0x4e/frame 0xfe00c6268950 > uma_dbg_free() at uma_dbg_free+0xfc/frame 0xfe00c62689a0 > item_dtor() at item_dtor+0x7c/frame 0xfe00c62689e0 > uma_zfree_arg() at uma_zfree_arg+0xf0/frame 0xfe00c6268a50 > uma_zfree() at uma_zfree+0x23/frame 0xfe00c6268a70 > xpt_free_ccb() at xpt_free_ccb+0x43/frame 0xfe00c6268a90 > camperiphdone() at camperiphdone+0x211/frame 0xfe00c6268ae0 > xpt_done_process() at xpt_done_process+0x550/frame 0xfe00c6268b40 > xpt_done_td() at xpt_done_td+0x1c0/frame 0xfe00c6268b80 > fork_exit() at fork_exit+0x117/frame 0xfe00c6268bf0 > fork_trampoline() at fork_trampoline+0xe/frame 0xfe00c6268bf0 > --- trap 0, rip = 0, rsp = 0, rbp = 0 --- > KDB: enter: panic > > doadump (textdump=0) at /usr/src/sys/kern/kern_shutdown.c:399 > 399 dumptid = curthread->td_tid; > (kgdb) bt > #0 doadump (textdump=0) at /usr/src/sys/kern/kern_shutdown.c:399 > #1 0x804d5dd7 in db_dump (dummy=-2138843371, dummy2=false, > dummy3=-1, > dummy4=0xfe00c6268320 "") at /usr/src/sys/ddb/db_command.c:575 > #2 0x804d5bf4 in db_command ( > last_cmdp=0x8114ce80 , cmd_table=0x0, > dopager=1) > at /usr/src/sys/ddb/db_command.c:482 > #3 0x804d583c in db_command_loop () > at /usr/src/sys/ddb/db_command.c:535 > #4 0x804da27c in db_trap (type=3, code=0) > at /usr/src/sys/ddb/db_main.c:270 > #5 0x8083df9d in kdb_trap (type=3, code=0, tf=0xfe00c6268770) > at /usr/src/sys/kern/subr_kdb.c:727 > #6 0x80d31494 in trap (frame=0xfe00c6268770) > at /usr/src/sys/amd64/amd64/trap.c:604 > #7 0x80d32628 in trap_check (frame=0xfe00c6268770) > at /usr/src/sys/amd64/amd64/trap.c:664 > #8 > #9 breakpoint () at /usr/src/sys/amd64/include/cpufunc.h:66 > #10 0x8083d3d0 in kdb_enter (why=0x80e0355b "panic", > msg=0x80e0355b "panic") at /usr/src/sys/kern/subr_kdb.c:505 > #11 0x807d1725 in vpanic ( > fmt=0x80dbca46 "Unaligned free of %p from zone %p(%s) slab > %p(%d)", ap=0xfe00c6268930) at /usr/src/sys/kern/kern_shutdown.c:906 > #12 0x807d120e in panic ( > fmt=0x80dbca46 "Unaligned free of %p from zone %p(%s) slab > %p(%d)") > at /usr/src/sys/kern/kern_shutdown.c:843 > #13 0x80c16a8c in uma_dbg_free (zone=0xfe00dc9d2000, > slab=0xf800259e2fd8, item=0xf800259e2800) > at /usr/src/sys/vm/uma_core.c:5659 > #14 0x80c0c5dc in item_dtor (zone=0xfe00dc9d2000, > item=0xf800259e2800, size=544, udata=0x0, skip=SKIP_NONE) > at /usr/src/sys/vm/uma_core.c:3418 > #15 0x80c0ba60 in uma_zfree_arg (zone=0xfe00dc9d2000, > item=0xf800259e2800, udata=0x0) at /usr/src/sys/vm/uma_core.c:4374 > #16 0x802e45d3 in uma_zfree (zone=0xfe00dc9d2000, > item=0xf800259e2800) at /usr/src/sys/vm/uma.h:404 > #17 0x802dc3c3 in xpt_free_ccb (free_ccb=0xf800259e2800) > at /usr/src/sys/cam/cam_xpt.c:4676 > #18 0x802dacf1 in camperiphdone (periph=0xf80025329b00, > done_ccb=0xf80025a24cc0) at /usr/src/sys/cam/cam_periph.c:1427 > #19 0x802e4520 in xpt_done_process (ccb_h=0xf80025a24cc0) > at /usr/src/sys/cam/cam_xpt.c:5493 > #20 0x802e68e0 in xpt_done_td (arg=0x81143700 ) > at /usr/src/sys/cam/cam_xpt.c:5548 > #21 0x807673c7 in fork_exit (callout=0x802e6720 > , > arg=0x81143700 , frame=0xfe00c6268c00) > at /usr/src/sys/kern/kern_fork.c:1083 > #22 > > [kgdb stuff removed] > > (kgdb) down > #15 0x80c0ba60 in uma_zfree_arg (zone=0xfe00dc9d2000, > item=0xf800259e2800, udata=0x0) at /usr/src/sys/vm/uma_core.c:4374 > 4374item_dtor(zone, item, cache_uz_size(cache), udata, > SKIP_NONE); > (kgdb) down > #14 0x80c0c5dc in item_dtor (zone=0xfe00dc9d2000, >
Re: panic: Unaligned free (was: kernel panic while copying files)
On Wed, 30 Jun 2021 06:02:59 +0100 Graham Perrin wrote: > On 29/06/2021 10:42, Gary Jennejohn wrote: > > ___ panic is now the result of an unaligned free. > > > > panic: Unaligned free of 0xf800259e2800 from zone > > 0xfe00dc9d2000(da_ccb) slab 0xf800259e2fd8(3) > > > > I have the crash dump and a debug kernel in case anyone wants more info. > Can you post the backtrace etc. here? Thanks > Sure. As can be seen from the uma zone being da_ccb, the panic resulted from setting kern.cam.da.enable_uma_ccbs=1. Unread portion of the kernel message buffer: panic: Unaligned free of 0xf800259e2800 from zone 0xfe00dc9d2000(da_ccb) slab 0xf800259e2fd8(3) cpuid = 2 time = 1624958650 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2c/frame 0xfe00c62687a0 kdb_backtrace() at kdb_backtrace+0x46/frame 0xfe00c6268850 vpanic() at vpanic+0x227/frame 0xfe00c62688f0 panic() at panic+0x4e/frame 0xfe00c6268950 uma_dbg_free() at uma_dbg_free+0xfc/frame 0xfe00c62689a0 item_dtor() at item_dtor+0x7c/frame 0xfe00c62689e0 uma_zfree_arg() at uma_zfree_arg+0xf0/frame 0xfe00c6268a50 uma_zfree() at uma_zfree+0x23/frame 0xfe00c6268a70 xpt_free_ccb() at xpt_free_ccb+0x43/frame 0xfe00c6268a90 camperiphdone() at camperiphdone+0x211/frame 0xfe00c6268ae0 xpt_done_process() at xpt_done_process+0x550/frame 0xfe00c6268b40 xpt_done_td() at xpt_done_td+0x1c0/frame 0xfe00c6268b80 fork_exit() at fork_exit+0x117/frame 0xfe00c6268bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfe00c6268bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- KDB: enter: panic doadump (textdump=0) at /usr/src/sys/kern/kern_shutdown.c:399 399 dumptid = curthread->td_tid; (kgdb) bt #0 doadump (textdump=0) at /usr/src/sys/kern/kern_shutdown.c:399 #1 0x804d5dd7 in db_dump (dummy=-2138843371, dummy2=false, dummy3=-1, dummy4=0xfe00c6268320 "") at /usr/src/sys/ddb/db_command.c:575 #2 0x804d5bf4 in db_command ( last_cmdp=0x8114ce80 , cmd_table=0x0, dopager=1) at /usr/src/sys/ddb/db_command.c:482 #3 0x804d583c in db_command_loop () at /usr/src/sys/ddb/db_command.c:535 #4 0x804da27c in db_trap (type=3, code=0) at /usr/src/sys/ddb/db_main.c:270 #5 0x8083df9d in kdb_trap (type=3, code=0, tf=0xfe00c6268770) at /usr/src/sys/kern/subr_kdb.c:727 #6 0x80d31494 in trap (frame=0xfe00c6268770) at /usr/src/sys/amd64/amd64/trap.c:604 #7 0x80d32628 in trap_check (frame=0xfe00c6268770) at /usr/src/sys/amd64/amd64/trap.c:664 #8 #9 breakpoint () at /usr/src/sys/amd64/include/cpufunc.h:66 #10 0x8083d3d0 in kdb_enter (why=0x80e0355b "panic", msg=0x80e0355b "panic") at /usr/src/sys/kern/subr_kdb.c:505 #11 0x807d1725 in vpanic ( fmt=0x80dbca46 "Unaligned free of %p from zone %p(%s) slab %p(%d)", ap=0xfe00c6268930) at /usr/src/sys/kern/kern_shutdown.c:906 #12 0x807d120e in panic ( fmt=0x80dbca46 "Unaligned free of %p from zone %p(%s) slab %p(%d)") at /usr/src/sys/kern/kern_shutdown.c:843 #13 0x80c16a8c in uma_dbg_free (zone=0xfe00dc9d2000, slab=0xf800259e2fd8, item=0xf800259e2800) at /usr/src/sys/vm/uma_core.c:5659 #14 0x80c0c5dc in item_dtor (zone=0xfe00dc9d2000, item=0xf800259e2800, size=544, udata=0x0, skip=SKIP_NONE) at /usr/src/sys/vm/uma_core.c:3418 #15 0x80c0ba60 in uma_zfree_arg (zone=0xfe00dc9d2000, item=0xf800259e2800, udata=0x0) at /usr/src/sys/vm/uma_core.c:4374 #16 0x802e45d3 in uma_zfree (zone=0xfe00dc9d2000, item=0xf800259e2800) at /usr/src/sys/vm/uma.h:404 #17 0x802dc3c3 in xpt_free_ccb (free_ccb=0xf800259e2800) at /usr/src/sys/cam/cam_xpt.c:4676 #18 0x802dacf1 in camperiphdone (periph=0xf80025329b00, done_ccb=0xf80025a24cc0) at /usr/src/sys/cam/cam_periph.c:1427 #19 0x802e4520 in xpt_done_process (ccb_h=0xf80025a24cc0) at /usr/src/sys/cam/cam_xpt.c:5493 #20 0x802e68e0 in xpt_done_td (arg=0x81143700 ) at /usr/src/sys/cam/cam_xpt.c:5548 #21 0x807673c7 in fork_exit (callout=0x802e6720 , arg=0x81143700 , frame=0xfe00c6268c00) at /usr/src/sys/kern/kern_fork.c:1083 #22 [kgdb stuff removed] (kgdb) down #15 0x80c0ba60 in uma_zfree_arg (zone=0xfe00dc9d2000, item=0xf800259e2800, udata=0x0) at /usr/src/sys/vm/uma_core.c:4374 4374item_dtor(zone, item, cache_uz_size(cache), udata, SKIP_NONE); (kgdb) down #14 0x80c0c5dc in item_dtor (zone=0xfe00dc9d2000, item=0xf800259e2800, size=544, udata=0x0, skip=SKIP_NONE) at /usr/src/sys/vm/uma_core.c:3418 3418uma_dbg_free(zone, NULL, item); (kgdb) p/x skipdbg $26 = 0x0 (kgdb) p/x zone->uz_flags $27 = 0x4100 (UMA_ZFLAG_TRASH|UMA_ZFLAG_CTORDTOR)
panic: Unaligned free (was: kernel panic while copying files)
On 29/06/2021 10:42, Gary Jennejohn wrote: … panic is now the result of an unaligned free. panic: Unaligned free of 0xf800259e2800 from zone 0xfe00dc9d2000(da_ccb) slab 0xf800259e2fd8(3) I have the crash dump and a debug kernel in case anyone wants more info. Can you post the backtrace etc. here? Thanks
