Re: panic(s) in ZFS on CURRENT
I will wait with my prepared merge until #14954 gets merged. On 9. 6. 2023 15:59, Alexander Motin wrote: Hi Gleb, There are two probably related PRs upstream: https://github.com/openzfs/zfs/pull/14939 https://github.com/openzfs/zfs/pull/14954 On 09.06.2023 00:57, Gleb Smirnoff wrote: On Thu, Jun 08, 2023 at 07:56:07PM -0700, Gleb Smirnoff wrote: T> I'm switching to INVARIANTS kernel right now and will see if that panics earlier. This is what I got with INVARIANTS: panic: VERIFY3(dev->l2ad_hand <= dev->l2ad_evict) failed (225142071296 <= 225142063104) cpuid = 17 time = 1686286015 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2c/frame 0xfe0160dcea90 kdb_backtrace() at kdb_backtrace+0x46/frame 0xfe0160dceb40 vpanic() at vpanic+0x21f/frame 0xfe0160dcebe0 spl_panic() at spl_panic+0x4d/frame 0xfe0160dcec60 l2arc_write_buffers() at l2arc_write_buffers+0xcda/frame 0xfe0160dcedf0 l2arc_feed_thread() at l2arc_feed_thread+0x547/frame 0xfe0160dceec0 fork_exit() at fork_exit+0x122/frame 0xfe0160dcef30 fork_trampoline() at fork_trampoline+0xe/frame 0xfe0160dcef30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Uptime: 1m4s Dumping 5473 out of 65308 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% (kgdb) frame 4 #4 0x804342ea in l2arc_write_buffers (spa=0xfe022e942000, dev=0xfe023116a000, target_sz=16777216) at /usr/src/FreeBSD/sys/contrib/openzfs/module/zfs/arc.c:9445 9445 ASSERT3U(dev->l2ad_hand, <=, dev->l2ad_evict); (kgdb) p dev $1 = (l2arc_dev_t *) 0xfe023116a000 (kgdb) p dev->l2ad_hand $2 = 225142071296 (kgdb) p dev->l2ad_evict $3 = 225142063104 (kgdb) p *dev value of type `l2arc_dev_t' requires 66136 bytes, which is more than max-value-size Never seen kgdb not being able to print a structure that reported to be too big.
Re: panic(s) in ZFS on CURRENT
Hi Gleb, There are two probably related PRs upstream: https://github.com/openzfs/zfs/pull/14939 https://github.com/openzfs/zfs/pull/14954 On 09.06.2023 00:57, Gleb Smirnoff wrote: On Thu, Jun 08, 2023 at 07:56:07PM -0700, Gleb Smirnoff wrote: T> I'm switching to INVARIANTS kernel right now and will see if that panics earlier. This is what I got with INVARIANTS: panic: VERIFY3(dev->l2ad_hand <= dev->l2ad_evict) failed (225142071296 <= 225142063104) cpuid = 17 time = 1686286015 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2c/frame 0xfe0160dcea90 kdb_backtrace() at kdb_backtrace+0x46/frame 0xfe0160dceb40 vpanic() at vpanic+0x21f/frame 0xfe0160dcebe0 spl_panic() at spl_panic+0x4d/frame 0xfe0160dcec60 l2arc_write_buffers() at l2arc_write_buffers+0xcda/frame 0xfe0160dcedf0 l2arc_feed_thread() at l2arc_feed_thread+0x547/frame 0xfe0160dceec0 fork_exit() at fork_exit+0x122/frame 0xfe0160dcef30 fork_trampoline() at fork_trampoline+0xe/frame 0xfe0160dcef30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Uptime: 1m4s Dumping 5473 out of 65308 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% (kgdb) frame 4 #4 0x804342ea in l2arc_write_buffers (spa=0xfe022e942000, dev=0xfe023116a000, target_sz=16777216) at /usr/src/FreeBSD/sys/contrib/openzfs/module/zfs/arc.c:9445 9445ASSERT3U(dev->l2ad_hand, <=, dev->l2ad_evict); (kgdb) p dev $1 = (l2arc_dev_t *) 0xfe023116a000 (kgdb) p dev->l2ad_hand $2 = 225142071296 (kgdb) p dev->l2ad_evict $3 = 225142063104 (kgdb) p *dev value of type `l2arc_dev_t' requires 66136 bytes, which is more than max-value-size Never seen kgdb not being able to print a structure that reported to be too big. -- Alexander Motin
Re: panic(s) in ZFS on CURRENT
On Thu, Jun 08, 2023 at 07:56:07PM -0700, Gleb Smirnoff wrote: T> I'm switching to INVARIANTS kernel right now and will see if that panics earlier. This is what I got with INVARIANTS: panic: VERIFY3(dev->l2ad_hand <= dev->l2ad_evict) failed (225142071296 <= 225142063104) cpuid = 17 time = 1686286015 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2c/frame 0xfe0160dcea90 kdb_backtrace() at kdb_backtrace+0x46/frame 0xfe0160dceb40 vpanic() at vpanic+0x21f/frame 0xfe0160dcebe0 spl_panic() at spl_panic+0x4d/frame 0xfe0160dcec60 l2arc_write_buffers() at l2arc_write_buffers+0xcda/frame 0xfe0160dcedf0 l2arc_feed_thread() at l2arc_feed_thread+0x547/frame 0xfe0160dceec0 fork_exit() at fork_exit+0x122/frame 0xfe0160dcef30 fork_trampoline() at fork_trampoline+0xe/frame 0xfe0160dcef30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Uptime: 1m4s Dumping 5473 out of 65308 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% (kgdb) frame 4 #4 0x804342ea in l2arc_write_buffers (spa=0xfe022e942000, dev=0xfe023116a000, target_sz=16777216) at /usr/src/FreeBSD/sys/contrib/openzfs/module/zfs/arc.c:9445 9445ASSERT3U(dev->l2ad_hand, <=, dev->l2ad_evict); (kgdb) p dev $1 = (l2arc_dev_t *) 0xfe023116a000 (kgdb) p dev->l2ad_hand $2 = 225142071296 (kgdb) p dev->l2ad_evict $3 = 225142063104 (kgdb) p *dev value of type `l2arc_dev_t' requires 66136 bytes, which is more than max-value-size Never seen kgdb not being able to print a structure that reported to be too big. -- Gleb Smirnoff
panic(s) in ZFS on CURRENT
Hi,
I got several panics on my desktop running eb2b00da564 which is
after the latest OpenZFS merge.
#1 (couple cores with this backtrace)
--- trap 0x9, rip = 0x803ab94b, rsp = 0xfe022e45ed30, rbp =
0xfe022e45ed50 ---
buf_hash_insert() at buf_hash_insert+0xab/frame 0xfe022e45ed50
arc_write_done() at arc_write_done+0xfa/frame 0xfe022e45ed90
zio_done() at zio_done+0xf0b/frame 0xfe022e45ee00
zio_execute() at zio_execute+0x9f/frame 0xfe022e45ee40
taskqueue_run_locked() at taskqueue_run_locked+0x181/frame 0xfe022e45eec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xc3/frame 0xfe022e45eef0
fork_exit() at fork_exit+0x7d/frame 0xfe022e45ef30
fork_trampoline() at fork_trampoline+0xe/frame 0xfe022e45ef30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
(kgdb) frame 7
#7 buf_hash_insert (hdr=hdr@entry=0xf8001b21fa28,
lockp=lockp@entry=0xfe022e45ed60)
at /usr/src/FreeBSD/sys/contrib/openzfs/module/zfs/arc.c:1062
1062if (HDR_EQUAL(hdr->b_spa, >b_dva, hdr->b_birth,
fhdr))
(kgdb) p hdr
$1 = (arc_buf_hdr_t *) 0xf8001b21fa28
(kgdb) p *hdr
$2 = {b_dva = {dva_word = {16, 20406677952}}, b_birth = 38447120, b_type =
ARC_BUFC_METADATA, b_complevel = 255 '\377', b_reserved1 = 0 '\000',
b_reserved2 = 0, b_hash_next = 0x0,
b_flags = (ARC_FLAG_L2CACHE | ARC_FLAG_IO_IN_PROGRESS |
ARC_FLAG_BUFC_METADATA | ARC_FLAG_HAS_L1HDR | ARC_FLAG_COMPRESSED_ARC |
ARC_FLAG_COMPRESS_0 | ARC_FLAG_COMPRESS_1 | ARC_FLAG_COMPRESS_2 |
ARC_FLAG_COMPRESS_3), b_psize = 8, b_lsize = 32, b_spa = 1230587331341359116,
b_l2hdr = {
b_dev = 0x0, b_daddr = 0, b_hits = 0, b_arcs_state = ARC_STATE_ANON,
b_l2node = {list_next = 0x0, list_prev = 0x0}}, b_l1hdr = {b_cv = {
cv_description = 0x80bb5b02 "hdr->b_l1hdr.b_cv", cv_waiters = 0},
b_byteswap = 10 '\n', b_state = 0x80ef23c0 ,
b_arc_node = {list_next = 0x0, list_prev = 0x0}, b_arc_access = 0,
b_mru_hits = 0, b_mru_ghost_hits = 0, b_mfu_hits = 0,
b_mfu_ghost_hits = 0, b_bufcnt = 1, b_buf = 0xf80003139d80, b_refcnt =
{rc_count = 2}, b_acb = 0x0, b_pabd = 0xf80a35dc6480},
b_crypt_hdr = {b_rabd = 0x10, b_ot = 2744191968, b_ebufcnt = 4, b_dsobj =
38340866, b_salt = "\001\000\000\000\000\000\000",
b_iv = "\000\000\000\000\000\000\000\000\220\000\026\017", b_mac = "\b\000
\000\f\230\262m\250\354\023\021\000\000\000"}}
#2 (single core)
--- trap 0x9, rip = 0x803ab94b, rsp = 0xfe0256158780, rbp =
0xfe02561587a0 ---
buf_hash_insert() at buf_hash_insert+0xab/frame 0xfe02561587a0
arc_hdr_realloc() at arc_hdr_realloc+0x138/frame 0xfe0256158800
arc_read() at arc_read+0x2dc/frame 0xfe02561588b0
dbuf_read() at dbuf_read+0xb3e/frame 0xfe02561589f0
dmu_buf_hold() at dmu_buf_hold+0x46/frame 0xfe0256158a30
zap_cursor_retrieve() at zap_cursor_retrieve+0x167/frame 0xfe0256158a90
zfs_freebsd_readdir() at zfs_freebsd_readdir+0x383/frame 0xfe0256158cc0
VOP_READDIR_APV() at VOP_READDIR_APV+0x1f/frame 0xfe0256158ce0
kern_getdirentries() at kern_getdirentries+0x186/frame 0xfe0256158dd0
sys_getdirentries() at sys_getdirentries+0x29/frame 0xfe0256158e00
amd64_syscall() at amd64_syscall+0x100/frame 0xfe0256158f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfe0256158f30
(kgdb) frame 7
#7 buf_hash_insert (hdr=0xf80c906b03e8, lockp=lockp@entry=0x0) at
/usr/src/FreeBSD/sys/contrib/openzfs/module/zfs/arc.c:1062
1062if (HDR_EQUAL(hdr->b_spa, >b_dva, hdr->b_birth,
fhdr))
(kgdb) p *hdr
$1 = {b_dva = {dva_word = {16, 19965896928}}, b_birth = 36629088, b_type =
ARC_BUFC_METADATA, b_complevel = 0 '\000', b_reserved1 = 0 '\000',
b_reserved2 = 0, b_hash_next = 0x0,
b_flags = (ARC_FLAG_BUFC_METADATA | ARC_FLAG_HAS_L1HDR | ARC_FLAG_HAS_L2HDR |
ARC_FLAG_COMPRESSED_ARC | ARC_FLAG_COMPRESS_1), b_psize = 5,
b_lsize = 5, b_spa = 3583499065027950438, b_l2hdr = {b_dev =
0xfe02306c8000, b_daddr = 4917395456, b_hits = 0,
b_arcs_state = ARC_STATE_MRU, b_l2node = {list_next = 0xf801313fc9b0,
list_prev = 0xf801313fca70}}, b_l1hdr = {b_cv = {
cv_description = 0x80bb5b02 "hdr->b_l1hdr.b_cv", cv_waiters = 0},
b_byteswap = 10 '\n',
b_state = 0x80f02900 , b_arc_node = {list_next = 0x0,
list_prev = 0x0}, b_arc_access = 0, b_mru_hits = 0,
b_mru_ghost_hits = 0, b_mfu_hits = 0, b_mfu_ghost_hits = 0, b_bufcnt = 0,
b_buf = 0x0, b_refcnt = {rc_count = 0}, b_acb = 0x0, b_pabd = 0x0},
b_crypt_hdr = {b_rabd = 0x10, b_ot = 2786027712, b_ebufcnt = 4, b_dsobj =
36629088, b_salt = "\001\000\000\000\000\000\000",
b_iv = "\240\2769$\001\370\377\377\220\000\036\017", b_mac = "\b\000
\000fw\357\327i%\2731\000\200l0"}}
#3 (not ZFS, but VFS, could be related?)
--- trap 0x9, rip = 0x80801408, rsp = 0xfe02348cbcc0, rbp =
0xfe02348cbcf0 ---
pwd_chdir() at pwd_chdir+0x28/frame 0xfe02348cbcf0
kern_chdir() at kern_chdir+0x169/frame
