Re: Repeatable panic on ZFS filesystem (used for backups); 11.0-STABLE

[Karl Denninger]

On 10/17/2016 18:32, Steven Hartland wrote:
>
> On 17/10/2016 22:50, Karl Denninger wrote:
>> I will make some effort on the sandbox machine to see if I can come up
>> with a way to replicate this.  I do have plenty of spare larger drives
>> laying around that used to be in service and were obsolesced due to
>> capacity -- but what I don't know if whether the system will misbehave
>> if the source is all spinning rust.
>>
>> In other words:
>>
>> 1. Root filesystem is mirrored spinning rust (production is mirrored
>> SSDs)
>>
>> 2. Backup is mirrored spinning rust (of approx the same size)
>>
>> 3. Set up auto-snapshot exactly as the production system has now (which
>> the sandbox is NOT since I don't care about incremental recovery on that
>> machine; it's a sandbox!)
>>
>> 4. Run a bunch of build-somethings (e.g. buildworlds, cross-build for
>> the Pi2s I have here, etc) to generate a LOT of filesystem entropy
>> across lots of snapshots.
>>
>> 5. Back that up.
>>
>> 6. Export the backup pool.
>>
>> 7. Re-import it and "zfs destroy -r" the backup filesystem.
>>
>> That is what got me in a reboot loop after the *first* panic; I was
>> simply going to destroy the backup filesystem and re-run the backup, but
>> as soon as I issued that zfs destroy the machine panic'd and as soon as
>> I re-attached it after a reboot it panic'd again.  Repeat until I set
>> trim=0.
>>
>> But... if I CAN replicate it that still shouldn't be happening, and the
>> system should *certainly* survive attempting to TRIM on a vdev that
>> doesn't support TRIMs, even if the removal is for a large amount of
>> space and/or files on the target, without blowing up.
>>
>> BTW I bet it isn't that rare -- if you're taking timed snapshots on an
>> active filesystem (with lots of entropy) and then make the mistake of
>> trying to remove those snapshots (as is the case with a zfs destroy -r
>> or a zfs recv of an incremental copy that attempts to sync against a
>> source) on a pool that has been imported before the system realizes that
>> TRIM is unavailable on those vdevs.
>>
>> Noting this:
>>
>>  Yes need to find some time to have a look at it, but given how rare
>>  this is and with TRIM being re-implemented upstream in a totally
>>  different manor I'm reticent to spend any real time on it.
>>
>> What's in-process in this regard, if you happen to have a reference?
> Looks like it may be still in review: https://reviews.csiden.org/r/263/
>
>

Having increased the kernel stack page count I have not had another
instance of this in the last couple of weeks+, and I am running daily
backup jobs as usual...

So this *does not* appear to be an infinite recursion problem...

-- 
Karl Denninger
k...@denninger.net 
/The Market Ticker/
/[S/MIME encrypted email preferred]/


smime.p7s
Description: S/MIME Cryptographic Signature

Re: Repeatable panic on ZFS filesystem (used for backups); 11.0-STABLE

[Karl Denninger]

On 10/17/2016 18:32, Steven Hartland wrote:
>
>
> On 17/10/2016 22:50, Karl Denninger wrote:
>> I will make some effort on the sandbox machine to see if I can come up
>> with a way to replicate this.  I do have plenty of spare larger drives
>> laying around that used to be in service and were obsolesced due to
>> capacity -- but what I don't know if whether the system will misbehave
>> if the source is all spinning rust.
>>
>> In other words:
>>
>> 1. Root filesystem is mirrored spinning rust (production is mirrored
>> SSDs)
>>
>> 2. Backup is mirrored spinning rust (of approx the same size)
>>
>> 3. Set up auto-snapshot exactly as the production system has now (which
>> the sandbox is NOT since I don't care about incremental recovery on that
>> machine; it's a sandbox!)
>>
>> 4. Run a bunch of build-somethings (e.g. buildworlds, cross-build for
>> the Pi2s I have here, etc) to generate a LOT of filesystem entropy
>> across lots of snapshots.
>>
>> 5. Back that up.
>>
>> 6. Export the backup pool.
>>
>> 7. Re-import it and "zfs destroy -r" the backup filesystem.
>>
>> That is what got me in a reboot loop after the *first* panic; I was
>> simply going to destroy the backup filesystem and re-run the backup, but
>> as soon as I issued that zfs destroy the machine panic'd and as soon as
>> I re-attached it after a reboot it panic'd again.  Repeat until I set
>> trim=0.
>>
>> But... if I CAN replicate it that still shouldn't be happening, and the
>> system should *certainly* survive attempting to TRIM on a vdev that
>> doesn't support TRIMs, even if the removal is for a large amount of
>> space and/or files on the target, without blowing up.
>>
>> BTW I bet it isn't that rare -- if you're taking timed snapshots on an
>> active filesystem (with lots of entropy) and then make the mistake of
>> trying to remove those snapshots (as is the case with a zfs destroy -r
>> or a zfs recv of an incremental copy that attempts to sync against a
>> source) on a pool that has been imported before the system realizes that
>> TRIM is unavailable on those vdevs.
>>
>> Noting this:
>>
>>  Yes need to find some time to have a look at it, but given how rare
>>  this is and with TRIM being re-implemented upstream in a totally
>>  different manor I'm reticent to spend any real time on it.
>>
>> What's in-process in this regard, if you happen to have a reference?
> Looks like it may be still in review: https://reviews.csiden.org/r/263/
>
>
Initial attempts to provoke the panic has failed on the sandbox machine
-- it appears that I need a materially-fragmented backup volume (which
makes sense, as that would greatly increase the number of TRIM's queued.)

Running a bunch of builds with snapshots taken between generates a
metric ton of entropy in the filesystem, but it appears that the number
of TRIMs actually issued when you bulk-remove them (with zfs destroy -r)
is small enough to not cause it -- probably because the system issues
one per area of freed disk, and since there is no interleaving with
other (non-removed) data that number is "reasonable" since there's
little fragmentation of that free space.

The TRIMs *are* attempted, and they *do* fail, however.

I'm running with the 6 pages of kstack now on the production machine,
and we'll see if I get another panic...

-- 
Karl Denninger
k...@denninger.net 
/The Market Ticker/
/[S/MIME encrypted email preferred]/


smime.p7s
Description: S/MIME Cryptographic Signature

Re: Repeatable panic on ZFS filesystem (used for backups); 11.0-STABLE

[Andriy Gapon]

On 18/10/2016 00:43, Steven Hartland wrote:
> On 17/10/2016 20:52, Andriy Gapon wrote:
>> On 17/10/2016 21:54, Steven Hartland wrote:
>>> You're hitting stack exhaustion, have you tried increasing the kernel stack 
>>> pages?
>>> It can be changed from /boot/loader.conf
>>> kern.kstack_pages="6"
>>>
>>> Default on amd64 is 4 IIRC
>> Steve,
>>
>> perhaps you can think of a more proper fix? :-)
>> https://lists.freebsd.org/pipermail/freebsd-stable/2016-July/085047.html
> Yes need to find some time to have a look at it, but given how rare this is 
> and
> with TRIM being re-implemented upstream in a totally different manor I'm
> reticent to spend any real time on it.

Fair enough.  Especially given that there is only a single affected system
reported so far.

-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Repeatable panic on ZFS filesystem (used for backups); 11.0-STABLE

[Steven Hartland]

On 17/10/2016 22:50, Karl Denninger wrote:

I will make some effort on the sandbox machine to see if I can come up
with a way to replicate this.  I do have plenty of spare larger drives
laying around that used to be in service and were obsolesced due to
capacity -- but what I don't know if whether the system will misbehave
if the source is all spinning rust.

In other words:

1. Root filesystem is mirrored spinning rust (production is mirrored SSDs)

2. Backup is mirrored spinning rust (of approx the same size)

3. Set up auto-snapshot exactly as the production system has now (which
the sandbox is NOT since I don't care about incremental recovery on that
machine; it's a sandbox!)

4. Run a bunch of build-somethings (e.g. buildworlds, cross-build for
the Pi2s I have here, etc) to generate a LOT of filesystem entropy
across lots of snapshots.

5. Back that up.

6. Export the backup pool.

7. Re-import it and "zfs destroy -r" the backup filesystem.

That is what got me in a reboot loop after the *first* panic; I was
simply going to destroy the backup filesystem and re-run the backup, but
as soon as I issued that zfs destroy the machine panic'd and as soon as
I re-attached it after a reboot it panic'd again.  Repeat until I set
trim=0.

But... if I CAN replicate it that still shouldn't be happening, and the
system should *certainly* survive attempting to TRIM on a vdev that
doesn't support TRIMs, even if the removal is for a large amount of
space and/or files on the target, without blowing up.

BTW I bet it isn't that rare -- if you're taking timed snapshots on an
active filesystem (with lots of entropy) and then make the mistake of
trying to remove those snapshots (as is the case with a zfs destroy -r
or a zfs recv of an incremental copy that attempts to sync against a
source) on a pool that has been imported before the system realizes that
TRIM is unavailable on those vdevs.

Noting this:

 Yes need to find some time to have a look at it, but given how rare
 this is and with TRIM being re-implemented upstream in a totally
 different manor I'm reticent to spend any real time on it.

What's in-process in this regard, if you happen to have a reference?

Looks like it may be still in review: https://reviews.csiden.org/r/263/

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Repeatable panic on ZFS filesystem (used for backups); 11.0-STABLE

[Karl Denninger]

I will make some effort on the sandbox machine to see if I can come up
with a way to replicate this.  I do have plenty of spare larger drives
laying around that used to be in service and were obsolesced due to
capacity -- but what I don't know if whether the system will misbehave
if the source is all spinning rust.

In other words:

1. Root filesystem is mirrored spinning rust (production is mirrored SSDs)

2. Backup is mirrored spinning rust (of approx the same size)

3. Set up auto-snapshot exactly as the production system has now (which
the sandbox is NOT since I don't care about incremental recovery on that
machine; it's a sandbox!)

4. Run a bunch of build-somethings (e.g. buildworlds, cross-build for
the Pi2s I have here, etc) to generate a LOT of filesystem entropy
across lots of snapshots.

5. Back that up.

6. Export the backup pool.

7. Re-import it and "zfs destroy -r" the backup filesystem.

That is what got me in a reboot loop after the *first* panic; I was
simply going to destroy the backup filesystem and re-run the backup, but
as soon as I issued that zfs destroy the machine panic'd and as soon as
I re-attached it after a reboot it panic'd again.  Repeat until I set
trim=0.

But... if I CAN replicate it that still shouldn't be happening, and the
system should *certainly* survive attempting to TRIM on a vdev that
doesn't support TRIMs, even if the removal is for a large amount of
space and/or files on the target, without blowing up.

BTW I bet it isn't that rare -- if you're taking timed snapshots on an
active filesystem (with lots of entropy) and then make the mistake of
trying to remove those snapshots (as is the case with a zfs destroy -r
or a zfs recv of an incremental copy that attempts to sync against a
source) on a pool that has been imported before the system realizes that
TRIM is unavailable on those vdevs.

Noting this:

Yes need to find some time to have a look at it, but given how rare
this is and with TRIM being re-implemented upstream in a totally
different manor I'm reticent to spend any real time on it.

What's in-process in this regard, if you happen to have a reference?

On 10/17/2016 16:40, Steven Hartland wrote:
> Setting those values will only effect what's queued to the device not
> what's actually outstanding.
>
> On 17/10/2016 21:22, Karl Denninger wrote:
>> Since I cleared it (by setting TRIM off on the test machine, rebooting,
>> importing the pool and noting that it did not panic -- pulled drives,
>> re-inserted into the production machine and ran backup routine -- all
>> was normal) it may be a while before I see it again (a week or so is
>> usual.)
>>
>> It appears to be related to entropy in the filesystem that comes up as
>> "eligible" to be removed from the backup volume, which (not
>> surprisingly) tends to happen a few days after I do a new world build or
>> something similar (the daily and/or periodic snapshots roll off at about
>> that point.)
>>
>> I don't happen to have a spare pair of high-performance SSDs I can stick
>> in the sandbox machine in an attempt to force the condition to assert
>> itself in test, unfortunately.
>>
>> I *am* concerned that it's not "simple" stack exhaustion because setting
>> the max outstanding TRIMs on a per-vdev basis down quite-dramatically
>> did *not* prevent it from happening -- and if it was simply stack depth
>> related I would have expected that to put a stop to it.
>>
>> On 10/17/2016 15:16, Steven Hartland wrote:
>>> Be good to confirm its not an infinite loop by giving it a good bump
>>> first.
>>>
>>> On 17/10/2016 19:58, Karl Denninger wrote:
 I can certainly attempt setting that higher but is that not just
 hiding the problem rather than addressing it?


 On 10/17/2016 13:54, Steven Hartland wrote:
> You're hitting stack exhaustion, have you tried increasing the kernel
> stack pages?
> It can be changed from /boot/loader.conf
> kern.kstack_pages="6"
>
> Default on amd64 is 4 IIRC
>
> On 17/10/2016 19:08, Karl Denninger wrote:
>> The target (and devices that trigger this) are a pair of 4Gb 7200RPM
>> SATA rotating rust drives (zmirror) with each provider
>> geli-encrypted
>> (that is, the actual devices used for the pool create are the
>> .eli's)
>>
>> The machine generating the problem has both rotating rust devices
>> *and*
>> SSDs, so I can't simply shut TRIM off system-wide and call it a
>> day as
>> TRIM itself is heavily-used; both the boot/root pools and a
>> Postgresql
>> database pool are on SSDs, while several terabytes of lesser-used
>> data
>> is on a pool of Raidz2 that is made up of spinning rust.
> snip...
>> NewFS.denninger.net dumped core - see /var/crash/vmcore.1
>>
>> Mon Oct 17 09:02:33 CDT 2016
>>
>> FreeBSD NewFS.denninger.net 11.0-STABLE FreeBSD 11.0-STABLE #13
>> r307318M: Fri Oct 14 09:23:46 CDT 2016
>> 

Re: Repeatable panic on ZFS filesystem (used for backups); 11.0-STABLE

[Steven Hartland]

On 17/10/2016 20:52, Andriy Gapon wrote:

On 17/10/2016 21:54, Steven Hartland wrote:

You're hitting stack exhaustion, have you tried increasing the kernel stack 
pages?
It can be changed from /boot/loader.conf
kern.kstack_pages="6"

Default on amd64 is 4 IIRC

Steve,

perhaps you can think of a more proper fix? :-)
https://lists.freebsd.org/pipermail/freebsd-stable/2016-July/085047.html
Yes need to find some time to have a look at it, but given how rare this 
is and with TRIM being re-implemented upstream in a totally different 
manor I'm reticent to spend any real time on it.

On 17/10/2016 19:08, Karl Denninger wrote:

The target (and devices that trigger this) are a pair of 4Gb 7200RPM
SATA rotating rust drives (zmirror) with each provider geli-encrypted
(that is, the actual devices used for the pool create are the .eli's)

The machine generating the problem has both rotating rust devices *and*
SSDs, so I can't simply shut TRIM off system-wide and call it a day as
TRIM itself is heavily-used; both the boot/root pools and a Postgresql
database pool are on SSDs, while several terabytes of lesser-used data
is on a pool of Raidz2 that is made up of spinning rust.

snip...

NewFS.denninger.net dumped core - see /var/crash/vmcore.1

Mon Oct 17 09:02:33 CDT 2016

FreeBSD NewFS.denninger.net 11.0-STABLE FreeBSD 11.0-STABLE #13
r307318M: Fri Oct 14 09:23:46 CDT 2016
k...@newfs.denninger.net:/usr/obj/usr/src/sys/KSD-SMP  amd64

panic: double fault

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:

Fatal double fault
rip = 0x8220d9ec
rsp = 0xfe066821f000
rbp = 0xfe066821f020
cpuid = 6; apic id = 14
panic: double fault
cpuid = 6
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
0xfe0649d78e30
vpanic() at vpanic+0x182/frame 0xfe0649d78eb0
panic() at panic+0x43/frame 0xfe0649d78f10
dblfault_handler() at dblfault_handler+0xa2/frame 0xfe0649d78f30
Xdblfault() at Xdblfault+0xac/frame 0xfe0649d78f30
--- trap 0x17, rip = 0x8220d9ec, rsp = 0xfe066821f000, rbp =
0xfe066821f020 ---
avl_rotation() at avl_rotation+0xc/frame 0xfe066821f020
avl_remove() at avl_remove+0x1c8/frame 0xfe066821f070
vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x87f/frame
0xfe066821f530
vdev_queue_io_done() at vdev_queue_io_done+0x83/frame 0xfe066821f570
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f5a0
zio_execute() at zio_execute+0x23d/frame 0xfe066821f5f0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821f650
zio_execute() at zio_execute+0x23d/frame 0xfe066821f6a0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821f6e0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f710
zio_execute() at zio_execute+0x23d/frame 0xfe066821f760
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821f7c0
zio_execute() at zio_execute+0x23d/frame 0xfe066821f810
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821f850
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f880
zio_execute() at zio_execute+0x23d/frame 0xfe066821f8d0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821f930
zio_execute() at zio_execute+0x23d/frame 0xfe066821f980
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821f9c0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f9f0
zio_execute() at zio_execute+0x23d/frame 0xfe066821fa40
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821faa0
zio_execute() at zio_execute+0x23d/frame 0xfe066821faf0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821fb30
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821fb60
zio_execute() at zio_execute+0x23d/frame 0xfe066821fbb0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821fc10
zio_execute() at zio_execute+0x23d/frame 0xfe066821fc60
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821fca0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821fcd0
zio_execute() at zio_execute+0x23d/frame 0xfe066821fd20
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821fd80
zio_execute() at zio_execute+0x23d/frame 0xfe066821fdd0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821fe10
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821fe40
zio_execute() at zio_execute+0x23d/frame 0xfe066821fe90
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821fef0
zio_execute() at zio_execute+0x23d/frame 

Re: Repeatable panic on ZFS filesystem (used for backups); 11.0-STABLE

[Steven Hartland]

Setting those values will only effect what's queued to the device not 
what's actually outstanding.


On 17/10/2016 21:22, Karl Denninger wrote:

Since I cleared it (by setting TRIM off on the test machine, rebooting,
importing the pool and noting that it did not panic -- pulled drives,
re-inserted into the production machine and ran backup routine -- all
was normal) it may be a while before I see it again (a week or so is usual.)

It appears to be related to entropy in the filesystem that comes up as
"eligible" to be removed from the backup volume, which (not
surprisingly) tends to happen a few days after I do a new world build or
something similar (the daily and/or periodic snapshots roll off at about
that point.)

I don't happen to have a spare pair of high-performance SSDs I can stick
in the sandbox machine in an attempt to force the condition to assert
itself in test, unfortunately.

I *am* concerned that it's not "simple" stack exhaustion because setting
the max outstanding TRIMs on a per-vdev basis down quite-dramatically
did *not* prevent it from happening -- and if it was simply stack depth
related I would have expected that to put a stop to it.

On 10/17/2016 15:16, Steven Hartland wrote:

Be good to confirm its not an infinite loop by giving it a good bump
first.

On 17/10/2016 19:58, Karl Denninger wrote:

I can certainly attempt setting that higher but is that not just
hiding the problem rather than addressing it?


On 10/17/2016 13:54, Steven Hartland wrote:

You're hitting stack exhaustion, have you tried increasing the kernel
stack pages?
It can be changed from /boot/loader.conf
kern.kstack_pages="6"

Default on amd64 is 4 IIRC

On 17/10/2016 19:08, Karl Denninger wrote:

The target (and devices that trigger this) are a pair of 4Gb 7200RPM
SATA rotating rust drives (zmirror) with each provider geli-encrypted
(that is, the actual devices used for the pool create are the .eli's)

The machine generating the problem has both rotating rust devices
*and*
SSDs, so I can't simply shut TRIM off system-wide and call it a day as
TRIM itself is heavily-used; both the boot/root pools and a Postgresql
database pool are on SSDs, while several terabytes of lesser-used data
is on a pool of Raidz2 that is made up of spinning rust.

snip...

NewFS.denninger.net dumped core - see /var/crash/vmcore.1

Mon Oct 17 09:02:33 CDT 2016

FreeBSD NewFS.denninger.net 11.0-STABLE FreeBSD 11.0-STABLE #13
r307318M: Fri Oct 14 09:23:46 CDT 2016
k...@newfs.denninger.net:/usr/obj/usr/src/sys/KSD-SMP  amd64

panic: double fault

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and
you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for
details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:

Fatal double fault
rip = 0x8220d9ec
rsp = 0xfe066821f000
rbp = 0xfe066821f020
cpuid = 6; apic id = 14
panic: double fault
cpuid = 6
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
0xfe0649d78e30
vpanic() at vpanic+0x182/frame 0xfe0649d78eb0
panic() at panic+0x43/frame 0xfe0649d78f10
dblfault_handler() at dblfault_handler+0xa2/frame 0xfe0649d78f30
Xdblfault() at Xdblfault+0xac/frame 0xfe0649d78f30
--- trap 0x17, rip = 0x8220d9ec, rsp = 0xfe066821f000,
rbp =
0xfe066821f020 ---
avl_rotation() at avl_rotation+0xc/frame 0xfe066821f020
avl_remove() at avl_remove+0x1c8/frame 0xfe066821f070
vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x87f/frame
0xfe066821f530
vdev_queue_io_done() at vdev_queue_io_done+0x83/frame
0xfe066821f570
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f5a0
zio_execute() at zio_execute+0x23d/frame 0xfe066821f5f0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfe066821f650
zio_execute() at zio_execute+0x23d/frame 0xfe066821f6a0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfe066821f6e0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f710
zio_execute() at zio_execute+0x23d/frame 0xfe066821f760
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfe066821f7c0
zio_execute() at zio_execute+0x23d/frame 0xfe066821f810
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfe066821f850
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f880
zio_execute() at zio_execute+0x23d/frame 0xfe066821f8d0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfe066821f930
zio_execute() at zio_execute+0x23d/frame 0xfe066821f980
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfe066821f9c0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f9f0
zio_execute() at zio_execute+0x23d/frame 0xfe066821fa40
zio_vdev_io_start() at 

Re: Repeatable panic on ZFS filesystem (used for backups); 11.0-STABLE

[Karl Denninger]

Since I cleared it (by setting TRIM off on the test machine, rebooting,
importing the pool and noting that it did not panic -- pulled drives,
re-inserted into the production machine and ran backup routine -- all
was normal) it may be a while before I see it again (a week or so is usual.)

It appears to be related to entropy in the filesystem that comes up as
"eligible" to be removed from the backup volume, which (not
surprisingly) tends to happen a few days after I do a new world build or
something similar (the daily and/or periodic snapshots roll off at about
that point.)

I don't happen to have a spare pair of high-performance SSDs I can stick
in the sandbox machine in an attempt to force the condition to assert
itself in test, unfortunately.

I *am* concerned that it's not "simple" stack exhaustion because setting
the max outstanding TRIMs on a per-vdev basis down quite-dramatically
did *not* prevent it from happening -- and if it was simply stack depth
related I would have expected that to put a stop to it.

On 10/17/2016 15:16, Steven Hartland wrote:
> Be good to confirm its not an infinite loop by giving it a good bump
> first.
>
> On 17/10/2016 19:58, Karl Denninger wrote:
>> I can certainly attempt setting that higher but is that not just
>> hiding the problem rather than addressing it?
>>
>>
>> On 10/17/2016 13:54, Steven Hartland wrote:
>>> You're hitting stack exhaustion, have you tried increasing the kernel
>>> stack pages?
>>> It can be changed from /boot/loader.conf
>>> kern.kstack_pages="6"
>>>
>>> Default on amd64 is 4 IIRC
>>>
>>> On 17/10/2016 19:08, Karl Denninger wrote:
 The target (and devices that trigger this) are a pair of 4Gb 7200RPM
 SATA rotating rust drives (zmirror) with each provider geli-encrypted
 (that is, the actual devices used for the pool create are the .eli's)

 The machine generating the problem has both rotating rust devices
 *and*
 SSDs, so I can't simply shut TRIM off system-wide and call it a day as
 TRIM itself is heavily-used; both the boot/root pools and a Postgresql
 database pool are on SSDs, while several terabytes of lesser-used data
 is on a pool of Raidz2 that is made up of spinning rust.
>>> snip...
 NewFS.denninger.net dumped core - see /var/crash/vmcore.1

 Mon Oct 17 09:02:33 CDT 2016

 FreeBSD NewFS.denninger.net 11.0-STABLE FreeBSD 11.0-STABLE #13
 r307318M: Fri Oct 14 09:23:46 CDT 2016
 k...@newfs.denninger.net:/usr/obj/usr/src/sys/KSD-SMP  amd64

 panic: double fault

 GNU gdb 6.1.1 [FreeBSD]
 Copyright 2004 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and
 you are
 welcome to change it and/or distribute copies of it under certain
 conditions.
 Type "show copying" to see the conditions.
 There is absolutely no warranty for GDB.  Type "show warranty" for
 details.
 This GDB was configured as "amd64-marcel-freebsd"...

 Unread portion of the kernel message buffer:

 Fatal double fault
 rip = 0x8220d9ec
 rsp = 0xfe066821f000
 rbp = 0xfe066821f020
 cpuid = 6; apic id = 14
 panic: double fault
 cpuid = 6
 KDB: stack backtrace:
 db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
 0xfe0649d78e30
 vpanic() at vpanic+0x182/frame 0xfe0649d78eb0
 panic() at panic+0x43/frame 0xfe0649d78f10
 dblfault_handler() at dblfault_handler+0xa2/frame 0xfe0649d78f30
 Xdblfault() at Xdblfault+0xac/frame 0xfe0649d78f30
 --- trap 0x17, rip = 0x8220d9ec, rsp = 0xfe066821f000,
 rbp =
 0xfe066821f020 ---
 avl_rotation() at avl_rotation+0xc/frame 0xfe066821f020
 avl_remove() at avl_remove+0x1c8/frame 0xfe066821f070
 vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x87f/frame
 0xfe066821f530
 vdev_queue_io_done() at vdev_queue_io_done+0x83/frame
 0xfe066821f570
 zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f5a0
 zio_execute() at zio_execute+0x23d/frame 0xfe066821f5f0
 zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
 0xfe066821f650
 zio_execute() at zio_execute+0x23d/frame 0xfe066821f6a0
 vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
 0xfe066821f6e0
 zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f710
 zio_execute() at zio_execute+0x23d/frame 0xfe066821f760
 zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
 0xfe066821f7c0
 zio_execute() at zio_execute+0x23d/frame 0xfe066821f810
 vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
 0xfe066821f850
 zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f880
 zio_execute() at zio_execute+0x23d/frame 0xfe066821f8d0
 zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
 0xfe066821f930
 zio_execute() at 

Re: Repeatable panic on ZFS filesystem (used for backups); 11.0-STABLE

[Steven Hartland]

Be good to confirm its not an infinite loop by giving it a good bump first.

On 17/10/2016 19:58, Karl Denninger wrote:

I can certainly attempt setting that higher but is that not just
hiding the problem rather than addressing it?


On 10/17/2016 13:54, Steven Hartland wrote:

You're hitting stack exhaustion, have you tried increasing the kernel
stack pages?
It can be changed from /boot/loader.conf
kern.kstack_pages="6"

Default on amd64 is 4 IIRC

On 17/10/2016 19:08, Karl Denninger wrote:

The target (and devices that trigger this) are a pair of 4Gb 7200RPM
SATA rotating rust drives (zmirror) with each provider geli-encrypted
(that is, the actual devices used for the pool create are the .eli's)

The machine generating the problem has both rotating rust devices *and*
SSDs, so I can't simply shut TRIM off system-wide and call it a day as
TRIM itself is heavily-used; both the boot/root pools and a Postgresql
database pool are on SSDs, while several terabytes of lesser-used data
is on a pool of Raidz2 that is made up of spinning rust.

snip...

NewFS.denninger.net dumped core - see /var/crash/vmcore.1

Mon Oct 17 09:02:33 CDT 2016

FreeBSD NewFS.denninger.net 11.0-STABLE FreeBSD 11.0-STABLE #13
r307318M: Fri Oct 14 09:23:46 CDT 2016
k...@newfs.denninger.net:/usr/obj/usr/src/sys/KSD-SMP  amd64

panic: double fault

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and
you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for
details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:

Fatal double fault
rip = 0x8220d9ec
rsp = 0xfe066821f000
rbp = 0xfe066821f020
cpuid = 6; apic id = 14
panic: double fault
cpuid = 6
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
0xfe0649d78e30
vpanic() at vpanic+0x182/frame 0xfe0649d78eb0
panic() at panic+0x43/frame 0xfe0649d78f10
dblfault_handler() at dblfault_handler+0xa2/frame 0xfe0649d78f30
Xdblfault() at Xdblfault+0xac/frame 0xfe0649d78f30
--- trap 0x17, rip = 0x8220d9ec, rsp = 0xfe066821f000, rbp =
0xfe066821f020 ---
avl_rotation() at avl_rotation+0xc/frame 0xfe066821f020
avl_remove() at avl_remove+0x1c8/frame 0xfe066821f070
vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x87f/frame
0xfe066821f530
vdev_queue_io_done() at vdev_queue_io_done+0x83/frame 0xfe066821f570
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f5a0
zio_execute() at zio_execute+0x23d/frame 0xfe066821f5f0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821f650
zio_execute() at zio_execute+0x23d/frame 0xfe066821f6a0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821f6e0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f710
zio_execute() at zio_execute+0x23d/frame 0xfe066821f760
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821f7c0
zio_execute() at zio_execute+0x23d/frame 0xfe066821f810
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821f850
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f880
zio_execute() at zio_execute+0x23d/frame 0xfe066821f8d0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821f930
zio_execute() at zio_execute+0x23d/frame 0xfe066821f980
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821f9c0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f9f0
zio_execute() at zio_execute+0x23d/frame 0xfe066821fa40
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821faa0
zio_execute() at zio_execute+0x23d/frame 0xfe066821faf0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821fb30
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821fb60
zio_execute() at zio_execute+0x23d/frame 0xfe066821fbb0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821fc10
zio_execute() at zio_execute+0x23d/frame 0xfe066821fc60
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821fca0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821fcd0
zio_execute() at zio_execute+0x23d/frame 0xfe066821fd20
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821fd80
zio_execute() at zio_execute+0x23d/frame 0xfe066821fdd0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821fe10
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821fe40
zio_execute() at zio_execute+0x23d/frame 0xfe066821fe90
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821fef0
zio_execute() at zio_execute+0x23d/frame 0xfe066821ff40
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821ff80
zio_vdev_io_done() at 

Re: Repeatable panic on ZFS filesystem (used for backups); 11.0-STABLE

[Andriy Gapon]

On 17/10/2016 21:54, Steven Hartland wrote:
> You're hitting stack exhaustion, have you tried increasing the kernel stack 
> pages?
> It can be changed from /boot/loader.conf
> kern.kstack_pages="6"
> 
> Default on amd64 is 4 IIRC

Steve,

perhaps you can think of a more proper fix? :-)
https://lists.freebsd.org/pipermail/freebsd-stable/2016-July/085047.html

> On 17/10/2016 19:08, Karl Denninger wrote:
>> The target (and devices that trigger this) are a pair of 4Gb 7200RPM
>> SATA rotating rust drives (zmirror) with each provider geli-encrypted
>> (that is, the actual devices used for the pool create are the .eli's)
>>
>> The machine generating the problem has both rotating rust devices *and*
>> SSDs, so I can't simply shut TRIM off system-wide and call it a day as
>> TRIM itself is heavily-used; both the boot/root pools and a Postgresql
>> database pool are on SSDs, while several terabytes of lesser-used data
>> is on a pool of Raidz2 that is made up of spinning rust.
> snip...
>>
>> NewFS.denninger.net dumped core - see /var/crash/vmcore.1
>>
>> Mon Oct 17 09:02:33 CDT 2016
>>
>> FreeBSD NewFS.denninger.net 11.0-STABLE FreeBSD 11.0-STABLE #13
>> r307318M: Fri Oct 14 09:23:46 CDT 2016
>> k...@newfs.denninger.net:/usr/obj/usr/src/sys/KSD-SMP  amd64
>>
>> panic: double fault
>>
>> GNU gdb 6.1.1 [FreeBSD]
>> Copyright 2004 Free Software Foundation, Inc.
>> GDB is free software, covered by the GNU General Public License, and you are
>> welcome to change it and/or distribute copies of it under certain
>> conditions.
>> Type "show copying" to see the conditions.
>> There is absolutely no warranty for GDB.  Type "show warranty" for details.
>> This GDB was configured as "amd64-marcel-freebsd"...
>>
>> Unread portion of the kernel message buffer:
>>
>> Fatal double fault
>> rip = 0x8220d9ec
>> rsp = 0xfe066821f000
>> rbp = 0xfe066821f020
>> cpuid = 6; apic id = 14
>> panic: double fault
>> cpuid = 6
>> KDB: stack backtrace:
>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
>> 0xfe0649d78e30
>> vpanic() at vpanic+0x182/frame 0xfe0649d78eb0
>> panic() at panic+0x43/frame 0xfe0649d78f10
>> dblfault_handler() at dblfault_handler+0xa2/frame 0xfe0649d78f30
>> Xdblfault() at Xdblfault+0xac/frame 0xfe0649d78f30
>> --- trap 0x17, rip = 0x8220d9ec, rsp = 0xfe066821f000, rbp =
>> 0xfe066821f020 ---
>> avl_rotation() at avl_rotation+0xc/frame 0xfe066821f020
>> avl_remove() at avl_remove+0x1c8/frame 0xfe066821f070
>> vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x87f/frame
>> 0xfe066821f530
>> vdev_queue_io_done() at vdev_queue_io_done+0x83/frame 0xfe066821f570
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f5a0
>> zio_execute() at zio_execute+0x23d/frame 0xfe066821f5f0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821f650
>> zio_execute() at zio_execute+0x23d/frame 0xfe066821f6a0
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821f6e0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f710
>> zio_execute() at zio_execute+0x23d/frame 0xfe066821f760
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821f7c0
>> zio_execute() at zio_execute+0x23d/frame 0xfe066821f810
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821f850
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f880
>> zio_execute() at zio_execute+0x23d/frame 0xfe066821f8d0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821f930
>> zio_execute() at zio_execute+0x23d/frame 0xfe066821f980
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821f9c0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f9f0
>> zio_execute() at zio_execute+0x23d/frame 0xfe066821fa40
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821faa0
>> zio_execute() at zio_execute+0x23d/frame 0xfe066821faf0
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821fb30
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821fb60
>> zio_execute() at zio_execute+0x23d/frame 0xfe066821fbb0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821fc10
>> zio_execute() at zio_execute+0x23d/frame 0xfe066821fc60
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821fca0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821fcd0
>> zio_execute() at zio_execute+0x23d/frame 0xfe066821fd20
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821fd80
>> zio_execute() at zio_execute+0x23d/frame 0xfe066821fdd0
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821fe10
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821fe40
>> zio_execute() at zio_execute+0x23d/frame 0xfe066821fe90
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821fef0
>> zio_execute() 

Re: Repeatable panic on ZFS filesystem (used for backups); 11.0-STABLE

[Karl Denninger]

I can certainly attempt setting that higher but is that not just
hiding the problem rather than addressing it?


On 10/17/2016 13:54, Steven Hartland wrote:
> You're hitting stack exhaustion, have you tried increasing the kernel
> stack pages?
> It can be changed from /boot/loader.conf
> kern.kstack_pages="6"
>
> Default on amd64 is 4 IIRC
>
> On 17/10/2016 19:08, Karl Denninger wrote:
>> The target (and devices that trigger this) are a pair of 4Gb 7200RPM
>> SATA rotating rust drives (zmirror) with each provider geli-encrypted
>> (that is, the actual devices used for the pool create are the .eli's)
>>
>> The machine generating the problem has both rotating rust devices *and*
>> SSDs, so I can't simply shut TRIM off system-wide and call it a day as
>> TRIM itself is heavily-used; both the boot/root pools and a Postgresql
>> database pool are on SSDs, while several terabytes of lesser-used data
>> is on a pool of Raidz2 that is made up of spinning rust.
> snip...
>>
>> NewFS.denninger.net dumped core - see /var/crash/vmcore.1
>>
>> Mon Oct 17 09:02:33 CDT 2016
>>
>> FreeBSD NewFS.denninger.net 11.0-STABLE FreeBSD 11.0-STABLE #13
>> r307318M: Fri Oct 14 09:23:46 CDT 2016
>> k...@newfs.denninger.net:/usr/obj/usr/src/sys/KSD-SMP  amd64
>>
>> panic: double fault
>>
>> GNU gdb 6.1.1 [FreeBSD]
>> Copyright 2004 Free Software Foundation, Inc.
>> GDB is free software, covered by the GNU General Public License, and
>> you are
>> welcome to change it and/or distribute copies of it under certain
>> conditions.
>> Type "show copying" to see the conditions.
>> There is absolutely no warranty for GDB.  Type "show warranty" for
>> details.
>> This GDB was configured as "amd64-marcel-freebsd"...
>>
>> Unread portion of the kernel message buffer:
>>
>> Fatal double fault
>> rip = 0x8220d9ec
>> rsp = 0xfe066821f000
>> rbp = 0xfe066821f020
>> cpuid = 6; apic id = 14
>> panic: double fault
>> cpuid = 6
>> KDB: stack backtrace:
>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
>> 0xfe0649d78e30
>> vpanic() at vpanic+0x182/frame 0xfe0649d78eb0
>> panic() at panic+0x43/frame 0xfe0649d78f10
>> dblfault_handler() at dblfault_handler+0xa2/frame 0xfe0649d78f30
>> Xdblfault() at Xdblfault+0xac/frame 0xfe0649d78f30
>> --- trap 0x17, rip = 0x8220d9ec, rsp = 0xfe066821f000, rbp =
>> 0xfe066821f020 ---
>> avl_rotation() at avl_rotation+0xc/frame 0xfe066821f020
>> avl_remove() at avl_remove+0x1c8/frame 0xfe066821f070
>> vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x87f/frame
>> 0xfe066821f530
>> vdev_queue_io_done() at vdev_queue_io_done+0x83/frame 0xfe066821f570
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f5a0
>> zio_execute() at zio_execute+0x23d/frame 0xfe066821f5f0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821f650
>> zio_execute() at zio_execute+0x23d/frame 0xfe066821f6a0
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821f6e0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f710
>> zio_execute() at zio_execute+0x23d/frame 0xfe066821f760
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821f7c0
>> zio_execute() at zio_execute+0x23d/frame 0xfe066821f810
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821f850
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f880
>> zio_execute() at zio_execute+0x23d/frame 0xfe066821f8d0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821f930
>> zio_execute() at zio_execute+0x23d/frame 0xfe066821f980
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821f9c0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f9f0
>> zio_execute() at zio_execute+0x23d/frame 0xfe066821fa40
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821faa0
>> zio_execute() at zio_execute+0x23d/frame 0xfe066821faf0
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821fb30
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821fb60
>> zio_execute() at zio_execute+0x23d/frame 0xfe066821fbb0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821fc10
>> zio_execute() at zio_execute+0x23d/frame 0xfe066821fc60
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821fca0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821fcd0
>> zio_execute() at zio_execute+0x23d/frame 0xfe066821fd20
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821fd80
>> zio_execute() at zio_execute+0x23d/frame 0xfe066821fdd0
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821fe10
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821fe40
>> zio_execute() at zio_execute+0x23d/frame 0xfe066821fe90
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821fef0
>> zio_execute() at 

Re: Repeatable panic on ZFS filesystem (used for backups); 11.0-STABLE

[Steven Hartland]

You're hitting stack exhaustion, have you tried increasing the kernel 
stack pages?

It can be changed from /boot/loader.conf
kern.kstack_pages="6"

Default on amd64 is 4 IIRC

On 17/10/2016 19:08, Karl Denninger wrote:

The target (and devices that trigger this) are a pair of 4Gb 7200RPM
SATA rotating rust drives (zmirror) with each provider geli-encrypted
(that is, the actual devices used for the pool create are the .eli's)

The machine generating the problem has both rotating rust devices *and*
SSDs, so I can't simply shut TRIM off system-wide and call it a day as
TRIM itself is heavily-used; both the boot/root pools and a Postgresql
database pool are on SSDs, while several terabytes of lesser-used data
is on a pool of Raidz2 that is made up of spinning rust.

snip...


NewFS.denninger.net dumped core - see /var/crash/vmcore.1

Mon Oct 17 09:02:33 CDT 2016

FreeBSD NewFS.denninger.net 11.0-STABLE FreeBSD 11.0-STABLE #13
r307318M: Fri Oct 14 09:23:46 CDT 2016
k...@newfs.denninger.net:/usr/obj/usr/src/sys/KSD-SMP  amd64

panic: double fault

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:

Fatal double fault
rip = 0x8220d9ec
rsp = 0xfe066821f000
rbp = 0xfe066821f020
cpuid = 6; apic id = 14
panic: double fault
cpuid = 6
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
0xfe0649d78e30
vpanic() at vpanic+0x182/frame 0xfe0649d78eb0
panic() at panic+0x43/frame 0xfe0649d78f10
dblfault_handler() at dblfault_handler+0xa2/frame 0xfe0649d78f30
Xdblfault() at Xdblfault+0xac/frame 0xfe0649d78f30
--- trap 0x17, rip = 0x8220d9ec, rsp = 0xfe066821f000, rbp =
0xfe066821f020 ---
avl_rotation() at avl_rotation+0xc/frame 0xfe066821f020
avl_remove() at avl_remove+0x1c8/frame 0xfe066821f070
vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x87f/frame
0xfe066821f530
vdev_queue_io_done() at vdev_queue_io_done+0x83/frame 0xfe066821f570
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f5a0
zio_execute() at zio_execute+0x23d/frame 0xfe066821f5f0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821f650
zio_execute() at zio_execute+0x23d/frame 0xfe066821f6a0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821f6e0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f710
zio_execute() at zio_execute+0x23d/frame 0xfe066821f760
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821f7c0
zio_execute() at zio_execute+0x23d/frame 0xfe066821f810
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821f850
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f880
zio_execute() at zio_execute+0x23d/frame 0xfe066821f8d0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821f930
zio_execute() at zio_execute+0x23d/frame 0xfe066821f980
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821f9c0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f9f0
zio_execute() at zio_execute+0x23d/frame 0xfe066821fa40
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821faa0
zio_execute() at zio_execute+0x23d/frame 0xfe066821faf0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821fb30
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821fb60
zio_execute() at zio_execute+0x23d/frame 0xfe066821fbb0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821fc10
zio_execute() at zio_execute+0x23d/frame 0xfe066821fc60
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821fca0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821fcd0
zio_execute() at zio_execute+0x23d/frame 0xfe066821fd20
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821fd80
zio_execute() at zio_execute+0x23d/frame 0xfe066821fdd0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821fe10
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821fe40
zio_execute() at zio_execute+0x23d/frame 0xfe066821fe90
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821fef0
zio_execute() at zio_execute+0x23d/frame 0xfe066821ff40
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821ff80
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821ffb0
zio_execute() at zio_execute+0x23d/frame 0xfe066822
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe0668220060
zio_execute() at zio_execute+0x23d/frame 0xfe06682200b0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 

Re: Repeatable panic on ZFS filesystem (used for backups); 11.0-STABLE

[Karl Denninger]

The target (and devices that trigger this) are a pair of 4Gb 7200RPM
SATA rotating rust drives (zmirror) with each provider geli-encrypted
(that is, the actual devices used for the pool create are the .eli's)

The machine generating the problem has both rotating rust devices *and*
SSDs, so I can't simply shut TRIM off system-wide and call it a day as
TRIM itself is heavily-used; both the boot/root pools and a Postgresql
database pool are on SSDs, while several terabytes of lesser-used data
is on a pool of Raidz2 that is made up of spinning rust.

vfs.zfs.trim.max_interval: 1
vfs.zfs.trim.timeout: 30
vfs.zfs.trim.txg_delay: 32
vfs.zfs.trim.enabled: 1
vfs.zfs.vdev.trim_max_pending: 1
vfs.zfs.vdev.trim_max_active: 64
vfs.zfs.vdev.trim_min_active: 1
vfs.zfs.vdev.trim_on_init: 1
kstat.zfs.misc.zio_trim.failed: 0
kstat.zfs.misc.zio_trim.unsupported: 1080
kstat.zfs.misc.zio_trim.success: 573768
kstat.zfs.misc.zio_trim.bytes: 28964282368

The machine in question has been up for ~3 hours now since the last
panic, so obviously TRIM is being heavily used...

The issue, once the problem has been created, is *portable* and it is
not being caused by the SSD source drives.  That is, once the machine
panics if I remove the two disks that form the backup pool, physically
move them to my sandbox machine, geli attach the drives and import the
pool within seconds the second machine will panic in the identical
fashion.  It's possible (but have not proved) that if I were to reboot
enough times the filesystem would eventually reach consistency with the
removed snapshots all gone and the panics would stop, but I got a
half-dozen of them sequentially this morning on my test machine so I'm
not at all sure how many more I'd need to allow to run, or whether *any*
of the removals committed before the panic (if not then the cycle of
reboot/attach/panic would never end) :-)

Reducing trim_max_active (to 10, a quite-drastic reduction) did not stop
the panics.

What appears to be happening is that the removal of the datasets in
question on a reasonably newly-imported pool, whether it occurs by the
incremental zfs recv -Fudv or by zfs destroy -r from the command line,
generates a large number of TRIM requests which are of course rejected
by the providers as spinning rust does not support them.  However the
attempt to queue them generates a stack overflow and double-fault panic
as a result, and since once the command is issued the filesystem now has
the deletes pending and the consistent state is in fact with them gone,
any attempt to reattach the drives with TRIM enabled can result in an
immediate additional panic.

I tried to work around this in my backup script by creating and then
destroying a file on the backup volume, then sleeping for a few seconds
before the backup actually commenced, in the hope that this would (1)
trigger a TRIM attempt and (2) lead the system to recognize that the
target volume cannot support TRIM and thus stop trying to do so (and
thus not lead to the storm that exhausts the stack and panic.)  That
approach, however (see below), failed to prevent the problem.

#
# Now try to trigger TRIM so that we don't have a storm of them
#
echo "Attempting to disable TRIM on spinning rust"

mount -t zfs backup/no-trim /mnt
dd if=/dev/random of=/mnt/kill-trim bs=128k count=2
echo "Performed 2 writes"
sleep 2
rm /mnt/kill-trim
echo "Performed delete of written file"
sleep 5
umount /mnt
echo "Unmounted temporary filesystem"
sleep 2
echo "TRIM disable theoretically done"


On 10/17/2016 12:43, Warner Losh wrote:
> what's your underlying media?
>
> Warner
>
>
> On Mon, Oct 17, 2016 at 10:02 AM, Karl Denninger  wrote:
>> Update from my test system:
>>
>> Setting vfs.zfs.vdev_trim_max_active to 10 (from default 64) does *not*
>> stop the panics.
>>
>> Setting vfs.zfs.vdev.trim.enabled = 0 (which requires a reboot) DOES
>> stop the panics.
>>
>> I am going to run a scrub on the pack, but I suspect the pack itself
>> (now that I can actually mount it without the machine blowing up!) is fine.
>>
>> THIS (OBVIOUSLY) NEEDS ATTENTION!
>>
>> On 10/17/2016 09:17, Karl Denninger wrote:
>>> This is a situation I've had happen before, and reported -- it appeared
>>> to be a kernel stack overflow, and it has gotten materially worse on
>>> 11.0-STABLE.
>>>
>>> The issue occurs after some period of time (normally a week or so.)  The
>>> system has a mirrored pair of large drives used for backup purposes to
>>> which ZFS snapshots are written using a script that iterates over the
>>> system.
>>>
>>> The panic /only /happens when the root filesystem is being sent, and it
>>> appears that the panic itself is being triggered by an I/O pattern on
>>> the /backup /drive -- not the source drives.  Zpool scrubs on the source
>>> are clean; I am going to run one now on the backup, but in the past that
>>> has been clean as well.
>>>
>>> I now have a *repeatable* panic in that if I attempt a "zfs list -rt all
>>> 

Re: Repeatable panic on ZFS filesystem (used for backups); 11.0-STABLE

[Warner Losh]

what's your underlying media?

Warner


On Mon, Oct 17, 2016 at 10:02 AM, Karl Denninger  wrote:
> Update from my test system:
>
> Setting vfs.zfs.vdev_trim_max_active to 10 (from default 64) does *not*
> stop the panics.
>
> Setting vfs.zfs.vdev.trim.enabled = 0 (which requires a reboot) DOES
> stop the panics.
>
> I am going to run a scrub on the pack, but I suspect the pack itself
> (now that I can actually mount it without the machine blowing up!) is fine.
>
> THIS (OBVIOUSLY) NEEDS ATTENTION!
>
> On 10/17/2016 09:17, Karl Denninger wrote:
>> This is a situation I've had happen before, and reported -- it appeared
>> to be a kernel stack overflow, and it has gotten materially worse on
>> 11.0-STABLE.
>>
>> The issue occurs after some period of time (normally a week or so.)  The
>> system has a mirrored pair of large drives used for backup purposes to
>> which ZFS snapshots are written using a script that iterates over the
>> system.
>>
>> The panic /only /happens when the root filesystem is being sent, and it
>> appears that the panic itself is being triggered by an I/O pattern on
>> the /backup /drive -- not the source drives.  Zpool scrubs on the source
>> are clean; I am going to run one now on the backup, but in the past that
>> has been clean as well.
>>
>> I now have a *repeatable* panic in that if I attempt a "zfs list -rt all
>> backup" on the backup volume I get the below panic.  A "zfs list"
>> does*__*not panic the system.
>>
>> The operating theory previously (after digging through the passed
>> structures in the dump) was that the ZFS system was attempting to issue
>> TRIMs on a device that can't do them before the ZFS system realizes this
>> and stops asking (the backup volume is comprised of spinning rust) but
>> the appearance of the panic now on the volume when I simply do a "zfs
>> list -rt all backup" appears to negate that theory since no writes are
>> performed by that operation, and thus no TRIM calls should be issued.
>>
>> I can leave the backup volume in the state that causes this for a short
>> period of time in an attempt to find and fix this.
>>
>>
>> NewFS.denninger.net dumped core - see /var/crash/vmcore.1
>>
>> Mon Oct 17 09:02:33 CDT 2016
>>
>> FreeBSD NewFS.denninger.net 11.0-STABLE FreeBSD 11.0-STABLE #13
>> r307318M: Fri Oct 14 09:23:46 CDT 2016
>> k...@newfs.denninger.net:/usr/obj/usr/src/sys/KSD-SMP  amd64
>>
>> panic: double fault
>>
>> GNU gdb 6.1.1 [FreeBSD]
>> Copyright 2004 Free Software Foundation, Inc.
>> GDB is free software, covered by the GNU General Public License, and you are
>> welcome to change it and/or distribute copies of it under certain
>> conditions.
>> Type "show copying" to see the conditions.
>> There is absolutely no warranty for GDB.  Type "show warranty" for details.
>> This GDB was configured as "amd64-marcel-freebsd"...
>>
>> Unread portion of the kernel message buffer:
>>
>> Fatal double fault
>> rip = 0x8220d9ec
>> rsp = 0xfe066821f000
>> rbp = 0xfe066821f020
>> cpuid = 6; apic id = 14
>> panic: double fault
>> cpuid = 6
>> KDB: stack backtrace:
>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
>> 0xfe0649d78e30
>> vpanic() at vpanic+0x182/frame 0xfe0649d78eb0
>> panic() at panic+0x43/frame 0xfe0649d78f10
>> dblfault_handler() at dblfault_handler+0xa2/frame 0xfe0649d78f30
>> Xdblfault() at Xdblfault+0xac/frame 0xfe0649d78f30
>> --- trap 0x17, rip = 0x8220d9ec, rsp = 0xfe066821f000, rbp =
>> 0xfe066821f020 ---
>> avl_rotation() at avl_rotation+0xc/frame 0xfe066821f020
>> avl_remove() at avl_remove+0x1c8/frame 0xfe066821f070
>> vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x87f/frame
>> 0xfe066821f530
>> vdev_queue_io_done() at vdev_queue_io_done+0x83/frame 0xfe066821f570
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f5a0
>> zio_execute() at zio_execute+0x23d/frame 0xfe066821f5f0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821f650
>> zio_execute() at zio_execute+0x23d/frame 0xfe066821f6a0
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821f6e0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f710
>> zio_execute() at zio_execute+0x23d/frame 0xfe066821f760
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821f7c0
>> zio_execute() at zio_execute+0x23d/frame 0xfe066821f810
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821f850
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f880
>> zio_execute() at zio_execute+0x23d/frame 0xfe066821f8d0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821f930
>> zio_execute() at zio_execute+0x23d/frame 0xfe066821f980
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821f9c0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f9f0
>> zio_execute() at zio_execute+0x23d/frame 0xfe066821fa40
>> 

Re: Repeatable panic on ZFS filesystem (used for backups); 11.0-STABLE

[Karl Denninger]

Update from my test system:

Setting vfs.zfs.vdev_trim_max_active to 10 (from default 64) does *not*
stop the panics.

Setting vfs.zfs.vdev.trim.enabled = 0 (which requires a reboot) DOES
stop the panics.

I am going to run a scrub on the pack, but I suspect the pack itself
(now that I can actually mount it without the machine blowing up!) is fine.

THIS (OBVIOUSLY) NEEDS ATTENTION!

On 10/17/2016 09:17, Karl Denninger wrote:
> This is a situation I've had happen before, and reported -- it appeared
> to be a kernel stack overflow, and it has gotten materially worse on
> 11.0-STABLE.
>
> The issue occurs after some period of time (normally a week or so.)  The
> system has a mirrored pair of large drives used for backup purposes to
> which ZFS snapshots are written using a script that iterates over the
> system.
>
> The panic /only /happens when the root filesystem is being sent, and it
> appears that the panic itself is being triggered by an I/O pattern on
> the /backup /drive -- not the source drives.  Zpool scrubs on the source
> are clean; I am going to run one now on the backup, but in the past that
> has been clean as well.
>
> I now have a *repeatable* panic in that if I attempt a "zfs list -rt all
> backup" on the backup volume I get the below panic.  A "zfs list"
> does*__*not panic the system.
>
> The operating theory previously (after digging through the passed
> structures in the dump) was that the ZFS system was attempting to issue
> TRIMs on a device that can't do them before the ZFS system realizes this
> and stops asking (the backup volume is comprised of spinning rust) but
> the appearance of the panic now on the volume when I simply do a "zfs
> list -rt all backup" appears to negate that theory since no writes are
> performed by that operation, and thus no TRIM calls should be issued.
>
> I can leave the backup volume in the state that causes this for a short
> period of time in an attempt to find and fix this.
>
>
> NewFS.denninger.net dumped core - see /var/crash/vmcore.1
>
> Mon Oct 17 09:02:33 CDT 2016
>
> FreeBSD NewFS.denninger.net 11.0-STABLE FreeBSD 11.0-STABLE #13
> r307318M: Fri Oct 14 09:23:46 CDT 2016
> k...@newfs.denninger.net:/usr/obj/usr/src/sys/KSD-SMP  amd64
>
> panic: double fault
>
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for details.
> This GDB was configured as "amd64-marcel-freebsd"...
>
> Unread portion of the kernel message buffer:
>
> Fatal double fault
> rip = 0x8220d9ec
> rsp = 0xfe066821f000
> rbp = 0xfe066821f020
> cpuid = 6; apic id = 14
> panic: double fault
> cpuid = 6
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
> 0xfe0649d78e30
> vpanic() at vpanic+0x182/frame 0xfe0649d78eb0
> panic() at panic+0x43/frame 0xfe0649d78f10
> dblfault_handler() at dblfault_handler+0xa2/frame 0xfe0649d78f30
> Xdblfault() at Xdblfault+0xac/frame 0xfe0649d78f30
> --- trap 0x17, rip = 0x8220d9ec, rsp = 0xfe066821f000, rbp =
> 0xfe066821f020 ---
> avl_rotation() at avl_rotation+0xc/frame 0xfe066821f020
> avl_remove() at avl_remove+0x1c8/frame 0xfe066821f070
> vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x87f/frame
> 0xfe066821f530
> vdev_queue_io_done() at vdev_queue_io_done+0x83/frame 0xfe066821f570
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f5a0
> zio_execute() at zio_execute+0x23d/frame 0xfe066821f5f0
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821f650
> zio_execute() at zio_execute+0x23d/frame 0xfe066821f6a0
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821f6e0
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f710
> zio_execute() at zio_execute+0x23d/frame 0xfe066821f760
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821f7c0
> zio_execute() at zio_execute+0x23d/frame 0xfe066821f810
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821f850
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f880
> zio_execute() at zio_execute+0x23d/frame 0xfe066821f8d0
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821f930
> zio_execute() at zio_execute+0x23d/frame 0xfe066821f980
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821f9c0
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f9f0
> zio_execute() at zio_execute+0x23d/frame 0xfe066821fa40
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821faa0
> zio_execute() at zio_execute+0x23d/frame 0xfe066821faf0
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821fb30
> 

Repeatable panic on ZFS filesystem (used for backups); 11.0-STABLE

[Karl Denninger]

This is a situation I've had happen before, and reported -- it appeared
to be a kernel stack overflow, and it has gotten materially worse on
11.0-STABLE.

The issue occurs after some period of time (normally a week or so.)  The
system has a mirrored pair of large drives used for backup purposes to
which ZFS snapshots are written using a script that iterates over the
system.

The panic /only /happens when the root filesystem is being sent, and it
appears that the panic itself is being triggered by an I/O pattern on
the /backup /drive -- not the source drives.  Zpool scrubs on the source
are clean; I am going to run one now on the backup, but in the past that
has been clean as well.

I now have a *repeatable* panic in that if I attempt a "zfs list -rt all
backup" on the backup volume I get the below panic.  A "zfs list"
does*__*not panic the system.

The operating theory previously (after digging through the passed
structures in the dump) was that the ZFS system was attempting to issue
TRIMs on a device that can't do them before the ZFS system realizes this
and stops asking (the backup volume is comprised of spinning rust) but
the appearance of the panic now on the volume when I simply do a "zfs
list -rt all backup" appears to negate that theory since no writes are
performed by that operation, and thus no TRIM calls should be issued.

I can leave the backup volume in the state that causes this for a short
period of time in an attempt to find and fix this.


NewFS.denninger.net dumped core - see /var/crash/vmcore.1

Mon Oct 17 09:02:33 CDT 2016

FreeBSD NewFS.denninger.net 11.0-STABLE FreeBSD 11.0-STABLE #13
r307318M: Fri Oct 14 09:23:46 CDT 2016
k...@newfs.denninger.net:/usr/obj/usr/src/sys/KSD-SMP  amd64

panic: double fault

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:

Fatal double fault
rip = 0x8220d9ec
rsp = 0xfe066821f000
rbp = 0xfe066821f020
cpuid = 6; apic id = 14
panic: double fault
cpuid = 6
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
0xfe0649d78e30
vpanic() at vpanic+0x182/frame 0xfe0649d78eb0
panic() at panic+0x43/frame 0xfe0649d78f10
dblfault_handler() at dblfault_handler+0xa2/frame 0xfe0649d78f30
Xdblfault() at Xdblfault+0xac/frame 0xfe0649d78f30
--- trap 0x17, rip = 0x8220d9ec, rsp = 0xfe066821f000, rbp =
0xfe066821f020 ---
avl_rotation() at avl_rotation+0xc/frame 0xfe066821f020
avl_remove() at avl_remove+0x1c8/frame 0xfe066821f070
vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x87f/frame
0xfe066821f530
vdev_queue_io_done() at vdev_queue_io_done+0x83/frame 0xfe066821f570
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f5a0
zio_execute() at zio_execute+0x23d/frame 0xfe066821f5f0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821f650
zio_execute() at zio_execute+0x23d/frame 0xfe066821f6a0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821f6e0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f710
zio_execute() at zio_execute+0x23d/frame 0xfe066821f760
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821f7c0
zio_execute() at zio_execute+0x23d/frame 0xfe066821f810
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821f850
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f880
zio_execute() at zio_execute+0x23d/frame 0xfe066821f8d0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821f930
zio_execute() at zio_execute+0x23d/frame 0xfe066821f980
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821f9c0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821f9f0
zio_execute() at zio_execute+0x23d/frame 0xfe066821fa40
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821faa0
zio_execute() at zio_execute+0x23d/frame 0xfe066821faf0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821fb30
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821fb60
zio_execute() at zio_execute+0x23d/frame 0xfe066821fbb0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821fc10
zio_execute() at zio_execute+0x23d/frame 0xfe066821fc60
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfe066821fca0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfe066821fcd0
zio_execute() at zio_execute+0x23d/frame 0xfe066821fd20
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfe066821fd80
zio_execute() at zio_execute+0x23d/frame 0xfe066821fdd0
vdev_queue_io_done() at