FreeBSD 12.0-STABLE #2 r349024M: Thu Jun 13 18:01:16 CDT 2019 k...@newfs.denninger.net:/usr/obj/usr/src/amd64.amd64/sys/KSD-SMP
Note -- no patches of any sort in the ZFS code; I am NOT running any of my former patch set. NewFS.denninger.net dumped core - see /var/crash/vmcore.8 Thu Jul 18 15:02:54 CDT 2019 FreeBSD NewFS.denninger.net 12.0-STABLE FreeBSD 12.0-STABLE #2 r349024M: Thu Jun 13 18:01:16 CDT 2019 k...@newfs.denninger.net:/usr/obj/usr/src/amd64.amd64/sys/KSD-SMP amd64 panic: double fault GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: Fatal double fault rip 0xffffffff8271eeec rsp 0xfffffe009d4e7f60 rbp 0xfffffe009d4e8450 rax 0xfffff801b5b68000 rdx 0xababe19 rbx 0xfffff801ac399000 rcx 0x6f598 rsi 0xfffff801b5b68740 rdi 0xfffff801ac2a2668 r8 0xfffff801ac2a2668 r9 0 r10 0xfffff801ac7cf250 r11 0 r12 0xfffff801b5b685b8 r13 0xfffff801b5b68000 r14 0xfffffe0082dfb000 r15 0xfffff801b5b685b8 rflags 0x10286 cs 0x20 ss 0x28 ds 0x3b es 0x3b fs 0x13 gs 0x1b fsbase 0x8002328d0 gsbase 0xffffffff8202a100 kgsbase 0 cpuid = 11; apic id = 35 panic: double fault cpuid = 11 time = 1563479881 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe000338edb0 vpanic() at vpanic+0x19d/frame 0xfffffe000338ee00 panic() at panic+0x43/frame 0xfffffe000338ee60 dblfault_handler() at dblfault_handler+0x1de/frame 0xfffffe000338ef30 Xdblfault() at Xdblfault+0xc3/frame 0xfffffe000338ef30 --- trap 0x17, rip = 0xffffffff8271eeec, rsp = 0xfffffe009d4e7f60, rbp = 0xfffffe009d4e8450 --- vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x2c/frame 0xfffffe009d4e8450 vdev_queue_io_done() at vdev_queue_io_done+0xc8/frame 0xfffffe009d4e84a0 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e84e0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e8530 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e8590 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e85e0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e8630 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e8670 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e86c0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e8720 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e8770 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e87c0 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e8800 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e8850 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e88b0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e8900 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e8950 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e8990 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e89e0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e8a40 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e8a90 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e8ae0 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e8b20 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e8b70 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e8bd0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e8c20 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e8c70 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e8cb0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e8d00 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e8d60 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e8db0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e8e00 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e8e40 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e8e90 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e8ef0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e8f40 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e8f90 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e8fd0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9020 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e9080 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e90d0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e9120 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e9160 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e91b0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e9210 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9260 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e92b0 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e92f0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9340 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e93a0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e93f0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e9440 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e9480 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e94d0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e9530 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9580 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e95d0 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e9610 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9660 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e96c0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9710 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e9760 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e97a0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e97f0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e9850 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e98a0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e98f0 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e9930 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9980 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e99e0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9a30 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e9a80 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e9ac0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9b10 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e9b70 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9bc0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e9c10 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e9c50 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9ca0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e9d00 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9d50 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e9da0 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e9de0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9e30 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e9e90 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9ee0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e9f30 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e9f70 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9fc0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ea020 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea070 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ea0c0 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ea100 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea150 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ea1b0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea200 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ea250 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ea290 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea2e0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ea340 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea390 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ea3e0 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ea420 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea470 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ea4d0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea520 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ea570 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ea5b0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea600 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ea660 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea6b0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ea700 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ea740 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea790 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ea7f0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea840 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ea890 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ea8d0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea920 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ea980 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea9d0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4eaa20 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4eaa60 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eaab0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4eab10 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eab60 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4eabb0 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4eabf0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eac40 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4eaca0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eacf0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ead40 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ead80 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eadd0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4eae30 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eae80 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4eaed0 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4eaf10 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eaf60 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4eafc0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb010 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4eb060 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4eb0a0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb0f0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4eb150 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb1a0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4eb1f0 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4eb230 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb280 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4eb2e0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb330 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4eb380 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4eb3c0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb410 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4eb470 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb4c0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4eb510 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4eb550 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb5a0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4eb600 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb650 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4eb6a0 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4eb6e0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb730 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4eb790 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb7e0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4eb830 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4eb870 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb8c0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4eb920 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb970 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4eb9c0 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4eba00 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eba50 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ebab0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ebb00 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ebb50 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ebb90 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ebbe0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ebc40 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ebc90 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ebce0 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ebd20 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ebd70 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ebdd0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ebe20 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ebe70 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ebeb0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ebf00 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ebf60 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ebfb0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ec000 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ec040 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec090 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ec0f0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec140 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ec190 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ec1d0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec220 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ec280 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec2d0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ec320 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ec360 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec3b0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ec410 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec460 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ec4b0 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ec4f0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec540 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ec5a0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec5f0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ec640 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ec680 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec6d0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ec730 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec780 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ec7d0 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ec810 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec860 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ec8c0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec910 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ec960 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ec9a0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec9f0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4eca50 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ecaa0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ecaf0 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ecb30 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ecb80 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ecbe0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ecc30 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ecc80 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4eccc0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ecd10 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ecd70 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ecdc0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ece10 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ece50 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ecea0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ecf00 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ecf50 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ecfa0 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ecfe0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed030 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ed090 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed0e0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ed130 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ed170 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed1c0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ed220 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed270 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ed2c0 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ed300 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed350 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ed3b0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed400 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ed450 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ed490 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed4e0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ed540 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed590 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ed5e0 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ed620 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed670 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ed6d0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed720 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ed770 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ed7b0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed800 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ed860 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed8b0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ed900 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ed940 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed990 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ed9f0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eda40 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4eda90 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4edad0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4edb20 taskqueue_run_locked() at taskqueue_run_locked+0x154/frame 0xfffffe009d4edb80 taskqueue_thread_loop() at taskqueue_thread_loop+0x98/frame 0xfffffe009d4edbb0 fork_exit() at fork_exit+0x83/frame 0xfffffe009d4edbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009d4edbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Uptime: 15m13s Dumping 1284 out of 12237 MB:..2%..12%..22%..32%..42%..52%..62%..72%..81%..91% (kgdb) #0 doadump (textdump=1) at src/sys/amd64/include/pcpu.h:234 #1 0xffffffff80bdb75d in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:451 #2 0xffffffff80bdbbe9 in vpanic (fmt=<value optimized out>, ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:877 #3 0xffffffff80bdb9e3 in panic (fmt=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:804 #4 0xffffffff810d904e in dblfault_handler (frame=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:1008 #5 0xffffffff810b27b3 in Xdblfault () at /usr/src/sys/amd64/amd64/exception.S:294 #6 0xffffffff8271eeec in vdev_queue_io_to_issue (vq=0xfffff801b5b685b8) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:638 #7 0xffffffff8271fb48 in vdev_queue_io_done (zio=0xfffff801ac399000) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:969 #8 0xffffffff82748d91 in zio_vdev_io_done (zio=0xfffff801ac399000) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3376 #9 0xffffffff82744eac in zio_execute (zio=0xfffff801ac399000) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1786 #10 0xffffffff82748ac7 in zio_vdev_io_start (zio=0xfffff801ac399000) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3350 #11 0xffffffff82744eac in zio_execute (zio=0xfffff801ac399000) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1786 #12 0xffffffff8271fb95 in vdev_queue_io_done (zio=<value optimized out>) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:975 #13 0xffffffff82748d91 in zio_vdev_io_done (zio=0xfffff8000b376830) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3376 #14 0xffffffff82744eac in zio_execute (zio=0xfffff8000b376830) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1786 #15 0xffffffff82748ac7 in zio_vdev_io_start (zio=0xfffff8000b376830) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3350 #16 0xffffffff82744eac in zio_execute (zio=0xfffff8000b376830) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1786 #17 0xffffffff8271fb95 in vdev_queue_io_done (zio=<value optimized out>) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:975 #18 0xffffffff82748d91 in zio_vdev_io_done (zio=0xfffff801ac744830) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3376 #19 0xffffffff82744eac in zio_execute (zio=0xfffff801ac744830) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1786 #20 0xffffffff82748ac7 in zio_vdev_io_start (zio=0xfffff801ac744830) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3350 #21 0xffffffff82744eac in zio_execute (zio=0xfffff801ac744830) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1786 #22 0xffffffff8271fb95 in vdev_queue_io_done (zio=<value optimized out>) ..... #275 0xffffffff82748ac7 in zio_vdev_io_start (zio=0xfffff8000b17e418) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3350 #276 0xffffffff82744eac in zio_execute (zio=0xfffff8000b17e418) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1786 #277 0xffffffff8271fb95 in vdev_queue_io_done (zio=<value optimized out>) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:975 #278 0xffffffff82748d91 in zio_vdev_io_done (zio=0xfffff801ac355000) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3376 #279 0xffffffff82744eac in zio_execute (zio=0xfffff801ac355000) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1786 #280 0xffffffff82748ac7 in zio_vdev_io_start (zio=0xfffff801ac355000) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3350 #281 0xffffffff82744eac in zio_execute (zio=0xfffff801ac355000) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1786 #282 0xffffffff8271fb95 in vdev_queue_io_done (zio=<value optimized out>) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:975 #283 0xffffffff82748d91 in zio_vdev_io_done (zio=0xfffff8000b8b8000) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3376 #284 0xffffffff82744eac in zio_execute (zio=0xfffff8000b8b8000) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1786 #285 0xffffffff80c3b7f4 in taskqueue_run_locked (queue=0xfffff801a8b35100) at /usr/src/sys/kern/subr_taskqueue.c:467 #286 0xffffffff80c3cb28 in taskqueue_thread_loop (arg=<value optimized out>) at /usr/src/sys/kern/subr_taskqueue.c:773 #287 0xffffffff80b9ab23 in fork_exit ( callout=0xffffffff80c3ca90 <taskqueue_thread_loop>, arg=0xfffff801a0577520, frame=0xfffffe009d4edc00) at /usr/src/sys/kern/kern_fork.c:1063 #288 0xffffffff810b367e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:996 #289 0x0000000000000000 in ?? () Current language: auto; currently minimal (kgdb) This is currently repeatable. What was going on at the instant in time was: root@NewFS:~ # /root/backup-zfs/run-backup Begin local ZFS backup by SEND Run backups of default [zsr/R/12.STABLE-2019-06-14 zsr/home zs/archive zs/colo-archive zs/disk zsr/dbms/pgsql zs/work zs/dbms/ticker-9.6] Thu Jul 18 14:57:57 CDT 2019 Import backup pool Imported; ready to proceed Processing zsr/R/12.STABLE-2019-06-14 Bring incremental backup up to date attempting destroy backup/R/12.STABLE-2019-06-14@zfs-auto-snap_daily-2019-07-10-00h07 success attempting destroy backup/R/12.STABLE-2019-06-14@zfs-auto-snap_daily-2019-07-11-00h07 success It destroyed the snapshot on the backup volume, and panic'd immediately thereafter. This is an incremental send. If I reboot the machine and re-start the backup job it will blow up when a couple more of the incremental deletes get done. Given the depth of the callback stack is this simply a kernel stack exhaustion problem? I wouldn't THINK it would be, but...... I do have this in /boot/loader.conf: # Try to avoid kernel stack exhaustion due to TRIM storms. kern.kstack_pages="6" The backup volumes are spinning rust, so there should be no "TRIM" attempts to them. In theory. I have the dump if someone wants me to run anything specific against it in terms of stack frames, etc. -- Karl Denninger k...@denninger.net <mailto:k...@denninger.net> /The Market Ticker/ /[S/MIME encrypted email preferred]/
smime.p7s
Description: S/MIME Cryptographic Signature