Re: Kernel panic in zfs code; 12-STABLE

2019-07-18 Thread Karl Denninger 
On 7/18/2019 15:35, Karl Denninger wrote:
> On 7/18/2019 15:19, Eugene Grosbein wrote:
>> 19.07.2019 3:13, Karl Denninger wrote:
>>
>>> FreeBSD 12.0-STABLE #2 r349024M: Thu Jun 13 18:01:16 CDT 2019
>>> k...@newfs.denninger.net:/usr/obj/usr/src/amd64.amd64/sys/KSD-SMP
>>>
>>> Note -- no patches of any sort in the ZFS code; I am NOT running any of
>>> my former patch set.
>>>
>>> NewFS.denninger.net dumped core - see /var/crash/vmcore.8
>>>
>>> Thu Jul 18 15:02:54 CDT 2019
>>>
>>> FreeBSD NewFS.denninger.net 12.0-STABLE FreeBSD 12.0-STABLE #2 r349024M:
>>> Thu Jun 13 18:01:16 CDT 2019
>>> k...@newfs.denninger.net:/usr/obj/usr/src/amd64.amd64/sys/KSD-SMP  amd64
>>>
>>> panic: double fault
>> [skip]
>>
>>> #283 0x82748d91 in zio_vdev_io_done (zio=0xf8000b8b8000)
>>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3376
>>> #284 0x82744eac in zio_execute (zio=0xf8000b8b8000)
>>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1786
>>> #285 0x80c3b7f4 in taskqueue_run_locked (queue=0xf801a8b35100)
>>> at /usr/src/sys/kern/subr_taskqueue.c:467
>>> #286 0x80c3cb28 in taskqueue_thread_loop (arg=)
>>> at /usr/src/sys/kern/subr_taskqueue.c:773
>>> #287 0x80b9ab23 in fork_exit (
>>> callout=0x80c3ca90 ,
>>> arg=0xf801a0577520, frame=0xfe009d4edc00)
>>> at /usr/src/sys/kern/kern_fork.c:1063
>>> #288 0x810b367e in fork_trampoline ()
>>> at /usr/src/sys/amd64/amd64/exception.S:996
>>> #289 0x in ?? ()
>>> Current language:  auto; currently minimal
>>> (kgdb)
>> You have "double fault" and completely insane number of stack frames in the 
>> trace.
>> This is obviously infinite recursion resulting in kernel stack overflow and 
>> panic.
> Yes, but why and how?
>
> What's executing at the time is this command:
>
> zfs send -RI $i@zfs-old $i@zfs-base | zfs receive -Fudv $BACKUP
>
> Which in turn results in the old snapshots on the target not on the
> source being deleted, then the new ones being sent.  It never gets to
> the sending part; it blows up during the delete of the OLD snapshots.
>
> The one(s) it deletes, however, it DOES delete.  When the box is
> rebooted those two snapshots on the target are indeed gone.
>
> That is, it is NOT getting "stuck" on one (which would imply there's an
> un-detected fault in the filesystem on the target in the metadata for
> that snapshot, resulting in a recursive call that blows up the stack)
> and it never gets to send the new snapshot, so whatever is going on is
> NOT on the source filesystem.  Neither source or destination shows any
> errors on the filesystem; both pools are healthy with zero error counts.
>
> Therefore the question -- is the system queueing enough work to blow the
> stack *BUT* the work it queues is all legitimate?  If so there's a
> serious problem in the way the code now functions in that an "ordinary"
> operation can result in what amounts to kernel stack exhaustion.
>
> One note -- I haven't run this backup for the last five days, as I do it
> manually and I've been out of town.  Previous running it on a daily
> basis completed without trouble.  This smells like a backlog of "things
> to do" when the send runs that results in the allegedly-infinite
> recursion (that isn't really infinite) that runs the stack out of space
> -- and THAT implies that the system is trying to queue a crazy amount of
> work on a recursive basis for what is a perfectly-legitimate operation
> -- which it should *NOT* do.

Update: This looks like an OLD bug that came back.

Previously the system would go absolutely insane on the first few
accesses to spinning rust during a snapshot delete and ATTEMPT to send
thousands of TRIM requests -- which spinning rust does not support.  On
a system with mixed vdevs, where some pools are rust and some are SSD,
this was a problem since you can't turn TRIM off because you REALLY want
it on those disks.

The FIX for this was to do this on the import of said pool comprised of
spinning rust:

#
# Now try to trigger TRIM so that we don't have a storm of them
#
# echo "Attempting to disable TRIM on spinning rust"

mount -t zfs $BACKUP/no-trim /mnt
dd if=/dev/random of=/mnt/kill-trim bs=128k count=2
echo "Performed 2 writes"
sleep 2
rm /mnt/kill-trim
echo "Performed delete of written file; wait"
sleep 35
umount /mnt
echo "Unmounted temporary filesystem"
sleep 2
echo "TRIM disable theoretically done"

This would cause ZFS to "figure out" that TRIM doesn't work on the VDEVs
on that pool and not send them.  You were then ok.

When this went away I commented those out.

I just un-commented them -- and the backup is now running, and appears
to be ok -- it's well past where it blew up before, and no panic.

So it APPEARS that ZFS (still) doesn't figure out that TRIM doesn't work
on those disks on a pool import quickly enough and this leads to a CRAZY
number of queued requests which

Re: Kernel panic in zfs code; 12-STABLE

2019-07-18 Thread Karl Denninger 
On 7/18/2019 15:19, Eugene Grosbein wrote:
> 19.07.2019 3:13, Karl Denninger wrote:
>
>> FreeBSD 12.0-STABLE #2 r349024M: Thu Jun 13 18:01:16 CDT 2019
>> k...@newfs.denninger.net:/usr/obj/usr/src/amd64.amd64/sys/KSD-SMP
>>
>> Note -- no patches of any sort in the ZFS code; I am NOT running any of
>> my former patch set.
>>
>> NewFS.denninger.net dumped core - see /var/crash/vmcore.8
>>
>> Thu Jul 18 15:02:54 CDT 2019
>>
>> FreeBSD NewFS.denninger.net 12.0-STABLE FreeBSD 12.0-STABLE #2 r349024M:
>> Thu Jun 13 18:01:16 CDT 2019
>> k...@newfs.denninger.net:/usr/obj/usr/src/amd64.amd64/sys/KSD-SMP  amd64
>>
>> panic: double fault
> [skip]
>
>> #283 0x82748d91 in zio_vdev_io_done (zio=0xf8000b8b8000)
>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3376
>> #284 0x82744eac in zio_execute (zio=0xf8000b8b8000)
>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1786
>> #285 0x80c3b7f4 in taskqueue_run_locked (queue=0xf801a8b35100)
>> at /usr/src/sys/kern/subr_taskqueue.c:467
>> #286 0x80c3cb28 in taskqueue_thread_loop (arg=)
>> at /usr/src/sys/kern/subr_taskqueue.c:773
>> #287 0x80b9ab23 in fork_exit (
>> callout=0x80c3ca90 ,
>> arg=0xf801a0577520, frame=0xfe009d4edc00)
>> at /usr/src/sys/kern/kern_fork.c:1063
>> #288 0x810b367e in fork_trampoline ()
>> at /usr/src/sys/amd64/amd64/exception.S:996
>> #289 0x in ?? ()
>> Current language:  auto; currently minimal
>> (kgdb)
> You have "double fault" and completely insane number of stack frames in the 
> trace.
> This is obviously infinite recursion resulting in kernel stack overflow and 
> panic.

Yes, but why and how?

What's executing at the time is this command:

zfs send -RI $i@zfs-old $i@zfs-base | zfs receive -Fudv $BACKUP

Which in turn results in the old snapshots on the target not on the
source being deleted, then the new ones being sent.  It never gets to
the sending part; it blows up during the delete of the OLD snapshots.

The one(s) it deletes, however, it DOES delete.  When the box is
rebooted those two snapshots on the target are indeed gone.

That is, it is NOT getting "stuck" on one (which would imply there's an
un-detected fault in the filesystem on the target in the metadata for
that snapshot, resulting in a recursive call that blows up the stack)
and it never gets to send the new snapshot, so whatever is going on is
NOT on the source filesystem.  Neither source or destination shows any
errors on the filesystem; both pools are healthy with zero error counts.

Therefore the question -- is the system queueing enough work to blow the
stack *BUT* the work it queues is all legitimate?  If so there's a
serious problem in the way the code now functions in that an "ordinary"
operation can result in what amounts to kernel stack exhaustion.

One note -- I haven't run this backup for the last five days, as I do it
manually and I've been out of town.  Previous running it on a daily
basis completed without trouble.  This smells like a backlog of "things
to do" when the send runs that results in the allegedly-infinite
recursion (that isn't really infinite) that runs the stack out of space
-- and THAT implies that the system is trying to queue a crazy amount of
work on a recursive basis for what is a perfectly-legitimate operation
-- which it should *NOT* do.

-- 
Karl Denninger
k...@denninger.net 
/The Market Ticker/
/[S/MIME encrypted email preferred]/


smime.p7s
Description: S/MIME Cryptographic Signature

Re: Kernel panic in zfs code; 12-STABLE

2019-07-18 Thread Eugene Grosbein 
19.07.2019 3:13, Karl Denninger wrote:

> FreeBSD 12.0-STABLE #2 r349024M: Thu Jun 13 18:01:16 CDT 2019
> k...@newfs.denninger.net:/usr/obj/usr/src/amd64.amd64/sys/KSD-SMP
> 
> Note -- no patches of any sort in the ZFS code; I am NOT running any of
> my former patch set.
> 
> NewFS.denninger.net dumped core - see /var/crash/vmcore.8
> 
> Thu Jul 18 15:02:54 CDT 2019
> 
> FreeBSD NewFS.denninger.net 12.0-STABLE FreeBSD 12.0-STABLE #2 r349024M:
> Thu Jun 13 18:01:16 CDT 2019
> k...@newfs.denninger.net:/usr/obj/usr/src/amd64.amd64/sys/KSD-SMP  amd64
> 
> panic: double fault

[skip]

> #283 0x82748d91 in zio_vdev_io_done (zio=0xf8000b8b8000)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3376
> #284 0x82744eac in zio_execute (zio=0xf8000b8b8000)
> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1786
> #285 0x80c3b7f4 in taskqueue_run_locked (queue=0xf801a8b35100)
> at /usr/src/sys/kern/subr_taskqueue.c:467
> #286 0x80c3cb28 in taskqueue_thread_loop (arg=)
> at /usr/src/sys/kern/subr_taskqueue.c:773
> #287 0x80b9ab23 in fork_exit (
> callout=0x80c3ca90 ,
> arg=0xf801a0577520, frame=0xfe009d4edc00)
> at /usr/src/sys/kern/kern_fork.c:1063
> #288 0x810b367e in fork_trampoline ()
> at /usr/src/sys/amd64/amd64/exception.S:996
> #289 0x in ?? ()
> Current language:  auto; currently minimal
> (kgdb)

You have "double fault" and completely insane number of stack frames in the 
trace.
This is obviously infinite recursion resulting in kernel stack overflow and 
panic.

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Kernel panic in zfs code; 12-STABLE

2019-07-18 Thread Karl Denninger 
FreeBSD 12.0-STABLE #2 r349024M: Thu Jun 13 18:01:16 CDT 2019
k...@newfs.denninger.net:/usr/obj/usr/src/amd64.amd64/sys/KSD-SMP

Note -- no patches of any sort in the ZFS code; I am NOT running any of
my former patch set.

NewFS.denninger.net dumped core - see /var/crash/vmcore.8

Thu Jul 18 15:02:54 CDT 2019

FreeBSD NewFS.denninger.net 12.0-STABLE FreeBSD 12.0-STABLE #2 r349024M:
Thu Jun 13 18:01:16 CDT 2019
k...@newfs.denninger.net:/usr/obj/usr/src/amd64.amd64/sys/KSD-SMP  amd64

panic: double fault

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:

Fatal double fault
rip 0x8271eeec rsp 0xfe009d4e7f60 rbp 0xfe009d4e8450
rax 0xf801b5b68000 rdx 0xababe19 rbx 0xf801ac399000
rcx 0x6f598 rsi 0xf801b5b68740 rdi 0xf801ac2a2668
r8 0xf801ac2a2668 r9 0 r10 0xf801ac7cf250
r11 0 r12 0xf801b5b685b8 r13 0xf801b5b68000
r14 0xfe0082dfb000 r15 0xf801b5b685b8 rflags 0x10286
cs 0x20 ss 0x28 ds 0x3b es 0x3b fs 0x13 gs 0x1b
fsbase 0x8002328d0 gsbase 0x8202a100 kgsbase 0
cpuid = 11; apic id = 35
panic: double fault
cpuid = 11
time = 1563479881
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
0xfe000338edb0
vpanic() at vpanic+0x19d/frame 0xfe000338ee00
panic() at panic+0x43/frame 0xfe000338ee60
dblfault_handler() at dblfault_handler+0x1de/frame 0xfe000338ef30
Xdblfault() at Xdblfault+0xc3/frame 0xfe000338ef30
--- trap 0x17, rip = 0x8271eeec, rsp = 0xfe009d4e7f60, rbp =
0xfe009d4e8450 ---
vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x2c/frame
0xfe009d4e8450
vdev_queue_io_done() at vdev_queue_io_done+0xc8/frame 0xfe009d4e84a0
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfe009d4e84e0
zio_execute() at zio_execute+0xac/frame 0xfe009d4e8530
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfe009d4e8590
zio_execute() at zio_execute+0xac/frame 0xfe009d4e85e0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfe009d4e8630
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfe009d4e8670
zio_execute() at zio_execute+0xac/frame 0xfe009d4e86c0
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfe009d4e8720
zio_execute() at zio_execute+0xac/frame 0xfe009d4e8770
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfe009d4e87c0
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfe009d4e8800
zio_execute() at zio_execute+0xac/frame 0xfe009d4e8850
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfe009d4e88b0
zio_execute() at zio_execute+0xac/frame 0xfe009d4e8900
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfe009d4e8950
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfe009d4e8990
zio_execute() at zio_execute+0xac/frame 0xfe009d4e89e0
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfe009d4e8a40
zio_execute() at zio_execute+0xac/frame 0xfe009d4e8a90
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfe009d4e8ae0
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfe009d4e8b20
zio_execute() at zio_execute+0xac/frame 0xfe009d4e8b70
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfe009d4e8bd0
zio_execute() at zio_execute+0xac/frame 0xfe009d4e8c20
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfe009d4e8c70
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfe009d4e8cb0
zio_execute() at zio_execute+0xac/frame 0xfe009d4e8d00
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfe009d4e8d60
zio_execute() at zio_execute+0xac/frame 0xfe009d4e8db0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfe009d4e8e00
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfe009d4e8e40
zio_execute() at zio_execute+0xac/frame 0xfe009d4e8e90
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfe009d4e8ef0
zio_execute() at zio_execute+0xac/frame 0xfe009d4e8f40
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfe009d4e8f90
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfe009d4e8fd0
zio_execute() at zio_execute+0xac/frame 0xfe009d4e9020
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfe009d4e9080
zio_execute() at zio_execute+0xac/frame 0xfe009d4e90d0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfe009d4e9120
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfe009d4e9160
zio_execute() at zio_execute+0xac/frame 0xfe009d4e91b0
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfe009d4e9210
zio_execute() at zio_execute+0xac/frame

Budget Cleaners Accounts Tax Invoice

2019-07-18 Thread Budget Cleaners ® 


 
You have received an Invoice
Budget Cleaners
from Budget Cleaners



View your invoice online:

https://accounting.sageone.co.za/customerzone/invoice/viewinvoice?TypeId=1=ffd62228-a6b3-4a8a-bb26-2ca3c5165bba=1=30491393


Dear Valued Client.
Thank You, for using Budget Cleaners, please kindly make payment for your Tax 
invoice as stated on the above link.
Please Note, all accounts are STRICTLY payable by the 20th of each month, 
unless prior arrangements have been made with the Accounts Dept.

Thank You.
Kind Regards.
Budget Cleaners Management.


Generated by Accounting

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: No sound in flashplayer with linux-c7

2019-07-18 Thread Serge Volkov via freebsd-stable 

18.07.2019 5:49, tech-lists пишет:
On Thu, Jul 18, 2019 at 02:14:57AM +0400, Serge Volkov via 
freebsd-stable wrote:

Hi, All!

I use www/flashplayer in Firefox on FreeBSD 12.0-RELEASE-p7 with KDE
Plasma 5. This week I updated the linux_base-c6 with all linux packages
to linux c7. After that, the sound disappeared in flashplayer. I also
rebuilt the flashplayer from the ports. But nothing has changed. No
sound. With the linux c6 the sound in flashplayer worked fine.

Can anyone help to solve problem with sound in flashplayer on linux c7?


Hi,

Does any sound work?

Do you have gsteamer plugins installed?


Yes, of course. Any sound works fine. In the firefox (Youtube and 
others) the sound works fine.


I even installed linux-c7-alsa-utils and tried play audio file with 
linux command line utility aplay in the KDE 5 Konsole


/compat/linux/usr/bin/aplay -vv audio.wav

and it's work fine, sound is good.

But there is no sound in the flashplayer only and only with linux-c7. 
With linux-c6 works fine.


I think it is related to the linux-c7-alsa-lib and/or 
linux-c7-alsa-plugins-oss. There may be a problem in its configuration 
files. But I do not know where exactly.


Or maybe this is related to the linux-flashplayer. Because the aplay 
utility works fine. I dont know :(


--
Regards,
Serge Volkov

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"