Re: Use after Free panic: ZFS?

2019-01-30 Thread Andriy Gapon 
On 29/01/2019 16:43, Larry Rosenman wrote:
> panic: Memory modified after free 0xf807019ca980(32) val=0 @ 
> 0xf807019ca980
> 
> cpuid = 5
> time = 1548755136
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfe00f750c880
> vpanic() at vpanic+0x1b4/frame 0xfe00f750c8e0
> panic() at panic+0x43/frame 0xfe00f750c940
> trash_ctor() at trash_ctor+0x4c/frame 0xfe00f750c950
> uma_zalloc_arg() at uma_zalloc_arg+0x9df/frame 0xfe00f750c9e0
> uma_zfree_arg() at uma_zfree_arg+0x46a/frame 0xfe00f750ca40
> arc_buf_destroy_impl() at arc_buf_destroy_impl+0x133/frame 0xfe00f750ca80
> arc_buf_destroy() at arc_buf_destroy+0x17a/frame 0xfe00f750cab0
> dbuf_destroy() at dbuf_destroy+0x87/frame 0xfe00f750cb10
> dbuf_evict_one() at dbuf_evict_one+0x187/frame 0xfe00f750cb40
> dbuf_evict_thread() at dbuf_evict_thread+0x185/frame 0xfe00f750cbb0
> fork_exit() at fork_exit+0x84/frame 0xfe00f750cbf0
> fork_trampoline() at fork_trampoline+0xe/frame 0xfe00f750cbf0
> --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
> Uptime: 3d16h49m14s
> Dumping 22587 out of 131028 
> MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
> 
> __curthread () at ./machine/pcpu.h:230
> 230 __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" 
> (OFFSETOF_CURTHREAD));
> (kgdb) #0  __curthread () at ./machine/pcpu.h:230
> #1  doadump (textdump=)
>     at /usr/src/sys/kern/kern_shutdown.c:371
> #2  0x80491760 in kern_reboot (howto=260)
>     at /usr/src/sys/kern/kern_shutdown.c:451
> #3  0x80491bc0 in vpanic (fmt=, ap=0xfe00f750c920)
>     at /usr/src/sys/kern/kern_shutdown.c:877
> #4  0x80491913 in panic (fmt=)
>     at /usr/src/sys/kern/kern_shutdown.c:804
> #5  0x8071255c in trash_ctor (mem=, size=,
>     arg=, flags=)
>     at /usr/src/sys/vm/uma_dbg.c:82
> #6  0x8070cf4f in uma_zalloc_arg (zone=0xf8203ffdc000,
>     udata=0x108, flags=1) at /usr/src/sys/vm/uma_core.c:2418
> #7  0x8070d69a in bucket_alloc (zone=,
>     udata=, flags=)
>     at /usr/src/sys/vm/uma_core.c:433
> #8  uma_zfree_arg (zone=0xf801059a, item=,
>     udata=0xf81042431940) at /usr/src/sys/vm/uma_core.c:3153

The problem is with an item in an (internal) UMA bucket zone.
So, this is probably not ZFS specific.

-- 
Andriy Gapon
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Use after Free panic: ZFS?

2019-01-29 Thread Larry Rosenman 

I've seen a couple of these...
⌂70% [l...@borg.lerctr.org:/var/crash] $ uname -aKU
FreeBSD borg.lerctr.org 13.0-CURRENT FreeBSD 13.0-CURRENT r343437 
LER-MINIMAL  amd64 139 139

⌂66% [l...@borg.lerctr.org:/var/crash] $

Ideas?  vmcore/symbols available.

borg.lerctr.org dumped core - see /var/crash/vmcore.7

Tue Jan 29 04:00:46 CST 2019

FreeBSD borg.lerctr.org 13.0-CURRENT FreeBSD 13.0-CURRENT r343437 
LER-MINIMAL  amd64


panic: Memory modified after free 0xf807019ca980(32) val=0 @ 
0xf807019ca980


GNU gdb (GDB) 8.2 [GDB v8.2 for FreeBSD]
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 


This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd13.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /boot/kernel/kernel...Reading symbols from 
/usr/lib/debug//boot/kernel/kernel.debug...done.

done.

Unread portion of the kernel message buffer:
panic: Memory modified after free 0xf807019ca980(32) val=0 @ 
0xf807019ca980


cpuid = 5
time = 1548755136
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 
0xfe00f750c880

vpanic() at vpanic+0x1b4/frame 0xfe00f750c8e0
panic() at panic+0x43/frame 0xfe00f750c940
trash_ctor() at trash_ctor+0x4c/frame 0xfe00f750c950
uma_zalloc_arg() at uma_zalloc_arg+0x9df/frame 0xfe00f750c9e0
uma_zfree_arg() at uma_zfree_arg+0x46a/frame 0xfe00f750ca40
arc_buf_destroy_impl() at arc_buf_destroy_impl+0x133/frame 
0xfe00f750ca80

arc_buf_destroy() at arc_buf_destroy+0x17a/frame 0xfe00f750cab0
dbuf_destroy() at dbuf_destroy+0x87/frame 0xfe00f750cb10
dbuf_evict_one() at dbuf_evict_one+0x187/frame 0xfe00f750cb40
dbuf_evict_thread() at dbuf_evict_thread+0x185/frame 0xfe00f750cbb0
fork_exit() at fork_exit+0x84/frame 0xfe00f750cbf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfe00f750cbf0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Uptime: 3d16h49m14s
Dumping 22587 out of 131028 
MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%


__curthread () at ./machine/pcpu.h:230
230 __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" 
(OFFSETOF_CURTHREAD));

(kgdb) #0  __curthread () at ./machine/pcpu.h:230
#1  doadump (textdump=)
at /usr/src/sys/kern/kern_shutdown.c:371
#2  0x80491760 in kern_reboot (howto=260)
at /usr/src/sys/kern/kern_shutdown.c:451
#3  0x80491bc0 in vpanic (fmt=, 
ap=0xfe00f750c920)

at /usr/src/sys/kern/kern_shutdown.c:877
#4  0x80491913 in panic (fmt=)
at /usr/src/sys/kern/kern_shutdown.c:804
#5  0x8071255c in trash_ctor (mem=, 
size=,

arg=, flags=)
at /usr/src/sys/vm/uma_dbg.c:82
#6  0x8070cf4f in uma_zalloc_arg (zone=0xf8203ffdc000,
udata=0x108, flags=1) at /usr/src/sys/vm/uma_core.c:2418
#7  0x8070d69a in bucket_alloc (zone=,
udata=, flags=)
at /usr/src/sys/vm/uma_core.c:433
#8  uma_zfree_arg (zone=0xf801059a, item=,
udata=0xf81042431940) at /usr/src/sys/vm/uma_core.c:3153
#9  0x812f8c13 in arc_free_data_buf (hdr=,
buf=0xfe025fe1e000, size=8192, tag=)
at 
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:5248

#10 arc_buf_destroy_impl (buf=0xf8190202ef00)
at 
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:3270

#11 0x812f859a in arc_buf_destroy (buf=0xf8190202ef00,
tag=0xf80aea618840)
at 
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:3687

#12 0x8130d3d7 in dbuf_destroy (db=0xf80aea618840)
at 
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:2328

#13 0x81313bb7 in dbuf_evict_one ()
at 
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:717

#14 0x8130b1d5 in dbuf_evict_thread (unused=)
at 
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:757

#15 0x80458a94 in fork_exit (
callout=0x8130b050 , arg=0x0,
frame=0xfe00f750cc00) at /usr/src/sys/kern/kern_fork.c:1055
#16 
(kgdb)



--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 214-642-9640 E-Mail: l...@lerctr.org
US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

use after free panic ZFS

2015-05-18 Thread Larry Rosenman 
found the following panic this am:

borg.lerctr.org dumped core - see /var/crash/vmcore.5

Sun May 17 23:47:48 CDT 2015

FreeBSD borg.lerctr.org 11.0-CURRENT FreeBSD 11.0-CURRENT #40 r283007: Sat May 
16 07:23:43 CDT 2015 r...@borg.lerctr.org:/usr/obj/usr/src/sys/VT-LER  amd64

panic: Most recently used by solaris

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type show copying to see the conditions.
There is absolutely no warranty for GDB.  Type show warranty for details.
This GDB was configured as amd64-marcel-freebsd...

Unread portion of the kernel message buffer:
Memory modified after free 0xf808535ea000(120) val=deadc0dd @ 
0xf808535ea050
panic: Most recently used by solaris

cpuid = 5
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfe100bfb7660
vpanic() at vpanic+0x189/frame 0xfe100bfb76e0
panic() at panic+0x43/frame 0xfe100bfb7740
mtrash_dtor() at mtrash_dtor/frame 0xfe100bfb7760
uma_zalloc_arg() at uma_zalloc_arg+0x4c2/frame 0xfe100bfb77d0
malloc() at malloc+0x198/frame 0xfe100bfb7820
zfs_range_lock() at zfs_range_lock+0x4a/frame 0xfe100bfb7880
zfs_get_data() at zfs_get_data+0x14c/frame 0xfe100bfb78f0
zil_commit() at zil_commit+0x94c/frame 0xfe100bfb7a10
zfs_freebsd_fsync() at zfs_freebsd_fsync+0xc8/frame 0xfe100bfb7a40
VOP_FSYNC_APV() at VOP_FSYNC_APV+0xf7/frame 0xfe100bfb7a70
sys_fsync() at sys_fsync+0x173/frame 0xfe100bfb7ae0
amd64_syscall() at amd64_syscall+0x25a/frame 0xfe100bfb7bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfe100bfb7bf0
--- syscall (95, FreeBSD ELF64, sys_fsync), rip = 0x801eb5daa, rsp = 
0x7fffd598, rbp = 0x7fffd5b0 ---
Uptime: 1d14h25m26s
Dumping 12469 out of 64457 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

Reading symbols from /boot/kernel/linux.ko.symbols...done.
Loaded symbols for /boot/kernel/linux.ko.symbols
Reading symbols from /boot/kernel/if_lagg.ko.symbols...done.
Loaded symbols for /boot/kernel/if_lagg.ko.symbols
Reading symbols from /boot/kernel/snd_envy24ht.ko.symbols...done.
Loaded symbols for /boot/kernel/snd_envy24ht.ko.symbols
Reading symbols from /boot/kernel/snd_spicds.ko.symbols...done.
Loaded symbols for /boot/kernel/snd_spicds.ko.symbols
Reading symbols from /boot/kernel/coretemp.ko.symbols...done.
Loaded symbols for /boot/kernel/coretemp.ko.symbols
Reading symbols from /boot/kernel/ichsmb.ko.symbols...done.
Loaded symbols for /boot/kernel/ichsmb.ko.symbols
Reading symbols from /boot/kernel/smbus.ko.symbols...done.
Loaded symbols for /boot/kernel/smbus.ko.symbols
Reading symbols from /boot/kernel/ichwd.ko.symbols...done.
Loaded symbols for /boot/kernel/ichwd.ko.symbols
Reading symbols from /boot/kernel/cpuctl.ko.symbols...done.
Loaded symbols for /boot/kernel/cpuctl.ko.symbols
Reading symbols from /boot/kernel/crypto.ko.symbols...done.
Loaded symbols for /boot/kernel/crypto.ko.symbols
Reading symbols from /boot/kernel/cryptodev.ko.symbols...done.
Loaded symbols for /boot/kernel/cryptodev.ko.symbols
Reading symbols from /boot/kernel/dtraceall.ko.symbols...done.
Loaded symbols for /boot/kernel/dtraceall.ko.symbols
Reading symbols from /boot/kernel/profile.ko.symbols...done.
Loaded symbols for /boot/kernel/profile.ko.symbols
Reading symbols from /boot/kernel/dtrace.ko.symbols...done.
Loaded symbols for /boot/kernel/dtrace.ko.symbols
Reading symbols from /boot/kernel/systrace_freebsd32.ko.symbols...done.
Loaded symbols for /boot/kernel/systrace_freebsd32.ko.symbols
Reading symbols from /boot/kernel/systrace.ko.symbols...done.
Loaded symbols for /boot/kernel/systrace.ko.symbols
Reading symbols from /boot/kernel/sdt.ko.symbols...done.
Loaded symbols for /boot/kernel/sdt.ko.symbols
Reading symbols from /boot/kernel/lockstat.ko.symbols...done.
Loaded symbols for /boot/kernel/lockstat.ko.symbols
Reading symbols from /boot/kernel/fasttrap.ko.symbols...done.
Loaded symbols for /boot/kernel/fasttrap.ko.symbols
Reading symbols from /boot/kernel/fbt.ko.symbols...done.
Loaded symbols for /boot/kernel/fbt.ko.symbols
Reading symbols from /boot/kernel/dtnfscl.ko.symbols...done.
Loaded symbols for /boot/kernel/dtnfscl.ko.symbols
Reading symbols from /boot/kernel/dtmalloc.ko.symbols...done.
Loaded symbols for /boot/kernel/dtmalloc.ko.symbols
Reading symbols from /boot/modules/vboxdrv.ko...done.
Loaded symbols for /boot/modules/vboxdrv.ko
Reading symbols from /boot/modules/nvidia.ko...done.
Loaded symbols for /boot/modules/nvidia.ko
Reading symbols from /boot/kernel/ipmi.ko.symbols...done.
Loaded symbols for /boot/kernel/ipmi.ko.symbols
Reading symbols from /boot/kernel/ipmi_linux.ko.symbols...done.
Loaded symbols for /boot/kernel/ipmi_linux.ko.symbols
Reading symbols from /boot/kernel/radeonkms.ko.symbols...done.
Loaded symbols for /boot/kernel/radeonkms.ko.symbols

Re: use after free panic ZFS

2015-05-18 Thread Larry Rosenman 
On Mon, May 18, 2015 at 07:42:47AM -0500, Larry Rosenman wrote:
 found the following panic this am:
 
 borg.lerctr.org dumped core - see /var/crash/vmcore.5
 
 Sun May 17 23:47:48 CDT 2015
 
 FreeBSD borg.lerctr.org 11.0-CURRENT FreeBSD 11.0-CURRENT #40 r283007: Sat 
 May 16 07:23:43 CDT 2015 r...@borg.lerctr.org:/usr/obj/usr/src/sys/VT-LER 
  amd64
 
 panic: Most recently used by solaris
 
 GNU gdb 6.1.1 [FreeBSD]
 Copyright 2004 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and you are
 welcome to change it and/or distribute copies of it under certain conditions.
 Type show copying to see the conditions.
 There is absolutely no warranty for GDB.  Type show warranty for details.
 This GDB was configured as amd64-marcel-freebsd...
 
 Unread portion of the kernel message buffer:
 Memory modified after free 0xf808535ea000(120) val=deadc0dd @ 
 0xf808535ea050
 panic: Most recently used by solaris
 
 cpuid = 5
 KDB: stack backtrace:
 db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfe100bfb7660
 vpanic() at vpanic+0x189/frame 0xfe100bfb76e0
 panic() at panic+0x43/frame 0xfe100bfb7740
 mtrash_dtor() at mtrash_dtor/frame 0xfe100bfb7760
 uma_zalloc_arg() at uma_zalloc_arg+0x4c2/frame 0xfe100bfb77d0
 malloc() at malloc+0x198/frame 0xfe100bfb7820
 zfs_range_lock() at zfs_range_lock+0x4a/frame 0xfe100bfb7880
 zfs_get_data() at zfs_get_data+0x14c/frame 0xfe100bfb78f0
 zil_commit() at zil_commit+0x94c/frame 0xfe100bfb7a10
 zfs_freebsd_fsync() at zfs_freebsd_fsync+0xc8/frame 0xfe100bfb7a40
 VOP_FSYNC_APV() at VOP_FSYNC_APV+0xf7/frame 0xfe100bfb7a70
 sys_fsync() at sys_fsync+0x173/frame 0xfe100bfb7ae0
 amd64_syscall() at amd64_syscall+0x25a/frame 0xfe100bfb7bf0
 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfe100bfb7bf0
 --- syscall (95, FreeBSD ELF64, sys_fsync), rip = 0x801eb5daa, rsp = 
 0x7fffd598, rbp = 0x7fffd5b0 ---
 Uptime: 1d14h25m26s
 Dumping 12469 out of 64457 
 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
 
 Reading symbols from /boot/kernel/linux.ko.symbols...done.
 Loaded symbols for /boot/kernel/linux.ko.symbols
 Reading symbols from /boot/kernel/if_lagg.ko.symbols...done.
 Loaded symbols for /boot/kernel/if_lagg.ko.symbols
 Reading symbols from /boot/kernel/snd_envy24ht.ko.symbols...done.
 Loaded symbols for /boot/kernel/snd_envy24ht.ko.symbols
 Reading symbols from /boot/kernel/snd_spicds.ko.symbols...done.
 Loaded symbols for /boot/kernel/snd_spicds.ko.symbols
 Reading symbols from /boot/kernel/coretemp.ko.symbols...done.
 Loaded symbols for /boot/kernel/coretemp.ko.symbols
 Reading symbols from /boot/kernel/ichsmb.ko.symbols...done.
 Loaded symbols for /boot/kernel/ichsmb.ko.symbols
 Reading symbols from /boot/kernel/smbus.ko.symbols...done.
 Loaded symbols for /boot/kernel/smbus.ko.symbols
 Reading symbols from /boot/kernel/ichwd.ko.symbols...done.
 Loaded symbols for /boot/kernel/ichwd.ko.symbols
 Reading symbols from /boot/kernel/cpuctl.ko.symbols...done.
 Loaded symbols for /boot/kernel/cpuctl.ko.symbols
 Reading symbols from /boot/kernel/crypto.ko.symbols...done.
 Loaded symbols for /boot/kernel/crypto.ko.symbols
 Reading symbols from /boot/kernel/cryptodev.ko.symbols...done.
 Loaded symbols for /boot/kernel/cryptodev.ko.symbols
 Reading symbols from /boot/kernel/dtraceall.ko.symbols...done.
 Loaded symbols for /boot/kernel/dtraceall.ko.symbols
 Reading symbols from /boot/kernel/profile.ko.symbols...done.
 Loaded symbols for /boot/kernel/profile.ko.symbols
 Reading symbols from /boot/kernel/dtrace.ko.symbols...done.
 Loaded symbols for /boot/kernel/dtrace.ko.symbols
 Reading symbols from /boot/kernel/systrace_freebsd32.ko.symbols...done.
 Loaded symbols for /boot/kernel/systrace_freebsd32.ko.symbols
 Reading symbols from /boot/kernel/systrace.ko.symbols...done.
 Loaded symbols for /boot/kernel/systrace.ko.symbols
 Reading symbols from /boot/kernel/sdt.ko.symbols...done.
 Loaded symbols for /boot/kernel/sdt.ko.symbols
 Reading symbols from /boot/kernel/lockstat.ko.symbols...done.
 Loaded symbols for /boot/kernel/lockstat.ko.symbols
 Reading symbols from /boot/kernel/fasttrap.ko.symbols...done.
 Loaded symbols for /boot/kernel/fasttrap.ko.symbols
 Reading symbols from /boot/kernel/fbt.ko.symbols...done.
 Loaded symbols for /boot/kernel/fbt.ko.symbols
 Reading symbols from /boot/kernel/dtnfscl.ko.symbols...done.
 Loaded symbols for /boot/kernel/dtnfscl.ko.symbols
 Reading symbols from /boot/kernel/dtmalloc.ko.symbols...done.
 Loaded symbols for /boot/kernel/dtmalloc.ko.symbols
 Reading symbols from /boot/modules/vboxdrv.ko...done.
 Loaded symbols for /boot/modules/vboxdrv.ko
 Reading symbols from /boot/modules/nvidia.ko...done.
 Loaded symbols for /boot/modules/nvidia.ko
 Reading symbols from /boot/kernel/ipmi.ko.symbols...done.
 Loaded symbols for /boot/kernel/ipmi.ko.symbols
 Reading symbols from /boot/kernel/ipmi_linux.ko.symbols...done.
 Loaded

Re: use after free panic ZFS

2015-05-18 Thread Larry Rosenman 
I've filed: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200288 for 
this.


I'd appreciate someone looking at it.

I'm STUCK here.



On 2015-05-18 07:56, Larry Rosenman wrote:

On Mon, May 18, 2015 at 07:42:47AM -0500, Larry Rosenman wrote:

found the following panic this am:

borg.lerctr.org dumped core - see /var/crash/vmcore.5

Sun May 17 23:47:48 CDT 2015

FreeBSD borg.lerctr.org 11.0-CURRENT FreeBSD 11.0-CURRENT #40 r283007: 
Sat May 16 07:23:43 CDT 2015 
r...@borg.lerctr.org:/usr/obj/usr/src/sys/VT-LER  amd64


panic: Most recently used by solaris

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and 
you are
welcome to change it and/or distribute copies of it under certain 
conditions.

Type show copying to see the conditions.
There is absolutely no warranty for GDB.  Type show warranty for 
details.

This GDB was configured as amd64-marcel-freebsd...

Unread portion of the kernel message buffer:
Memory modified after free 0xf808535ea000(120) val=deadc0dd @ 
0xf808535ea050

panic: Most recently used by solaris

cpuid = 5
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 
0xfe100bfb7660

vpanic() at vpanic+0x189/frame 0xfe100bfb76e0
panic() at panic+0x43/frame 0xfe100bfb7740
mtrash_dtor() at mtrash_dtor/frame 0xfe100bfb7760
uma_zalloc_arg() at uma_zalloc_arg+0x4c2/frame 0xfe100bfb77d0
malloc() at malloc+0x198/frame 0xfe100bfb7820
zfs_range_lock() at zfs_range_lock+0x4a/frame 0xfe100bfb7880
zfs_get_data() at zfs_get_data+0x14c/frame 0xfe100bfb78f0
zil_commit() at zil_commit+0x94c/frame 0xfe100bfb7a10
zfs_freebsd_fsync() at zfs_freebsd_fsync+0xc8/frame 0xfe100bfb7a40
VOP_FSYNC_APV() at VOP_FSYNC_APV+0xf7/frame 0xfe100bfb7a70
sys_fsync() at sys_fsync+0x173/frame 0xfe100bfb7ae0
amd64_syscall() at amd64_syscall+0x25a/frame 0xfe100bfb7bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfe100bfb7bf0
--- syscall (95, FreeBSD ELF64, sys_fsync), rip = 0x801eb5daa, rsp = 
0x7fffd598, rbp = 0x7fffd5b0 ---

Uptime: 1d14h25m26s
Dumping 12469 out of 64457 
MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%


Reading symbols from /boot/kernel/linux.ko.symbols...done.
Loaded symbols for /boot/kernel/linux.ko.symbols
Reading symbols from /boot/kernel/if_lagg.ko.symbols...done.
Loaded symbols for /boot/kernel/if_lagg.ko.symbols
Reading symbols from /boot/kernel/snd_envy24ht.ko.symbols...done.
Loaded symbols for /boot/kernel/snd_envy24ht.ko.symbols
Reading symbols from /boot/kernel/snd_spicds.ko.symbols...done.
Loaded symbols for /boot/kernel/snd_spicds.ko.symbols
Reading symbols from /boot/kernel/coretemp.ko.symbols...done.
Loaded symbols for /boot/kernel/coretemp.ko.symbols
Reading symbols from /boot/kernel/ichsmb.ko.symbols...done.
Loaded symbols for /boot/kernel/ichsmb.ko.symbols
Reading symbols from /boot/kernel/smbus.ko.symbols...done.
Loaded symbols for /boot/kernel/smbus.ko.symbols
Reading symbols from /boot/kernel/ichwd.ko.symbols...done.
Loaded symbols for /boot/kernel/ichwd.ko.symbols
Reading symbols from /boot/kernel/cpuctl.ko.symbols...done.
Loaded symbols for /boot/kernel/cpuctl.ko.symbols
Reading symbols from /boot/kernel/crypto.ko.symbols...done.
Loaded symbols for /boot/kernel/crypto.ko.symbols
Reading symbols from /boot/kernel/cryptodev.ko.symbols...done.
Loaded symbols for /boot/kernel/cryptodev.ko.symbols
Reading symbols from /boot/kernel/dtraceall.ko.symbols...done.
Loaded symbols for /boot/kernel/dtraceall.ko.symbols
Reading symbols from /boot/kernel/profile.ko.symbols...done.
Loaded symbols for /boot/kernel/profile.ko.symbols
Reading symbols from /boot/kernel/dtrace.ko.symbols...done.
Loaded symbols for /boot/kernel/dtrace.ko.symbols
Reading symbols from 
/boot/kernel/systrace_freebsd32.ko.symbols...done.

Loaded symbols for /boot/kernel/systrace_freebsd32.ko.symbols
Reading symbols from /boot/kernel/systrace.ko.symbols...done.
Loaded symbols for /boot/kernel/systrace.ko.symbols
Reading symbols from /boot/kernel/sdt.ko.symbols...done.
Loaded symbols for /boot/kernel/sdt.ko.symbols
Reading symbols from /boot/kernel/lockstat.ko.symbols...done.
Loaded symbols for /boot/kernel/lockstat.ko.symbols
Reading symbols from /boot/kernel/fasttrap.ko.symbols...done.
Loaded symbols for /boot/kernel/fasttrap.ko.symbols
Reading symbols from /boot/kernel/fbt.ko.symbols...done.
Loaded symbols for /boot/kernel/fbt.ko.symbols
Reading symbols from /boot/kernel/dtnfscl.ko.symbols...done.
Loaded symbols for /boot/kernel/dtnfscl.ko.symbols
Reading symbols from /boot/kernel/dtmalloc.ko.symbols...done.
Loaded symbols for /boot/kernel/dtmalloc.ko.symbols
Reading symbols from /boot/modules/vboxdrv.ko...done.
Loaded symbols for /boot/modules/vboxdrv.ko
Reading symbols from /boot/modules/nvidia.ko...done.
Loaded symbols for /boot/modules/nvidia.ko
Reading symbols from /boot/kernel/ipmi.ko.symbols...done.
Loaded symbols for