@GernotS I got the dump from your latest panic (*.37) and I'm still not sure
what's going on there. I am running some stress tests with encryption and
l2arc and haven't hit a panic yet. Is there any chance you can reproduce with
debug bits? Even just debug zfs kernel module would be helpful since then
assertions would be enabled.
We are evicting a hdr, and it points to a buf that has been freed. The hdr is
for a block of dnodes from the MOS (it's not encrypted/protected). The hdr is
in the l2arc (`HAS_L2HDR`). The only place that the buf is freed is
`arc_buf_destroy_impl()`. The `hash_lock` protects the hdr from changing
while we remove the buf from the linked list and free it.
The PREDICTIVE_PREFETCH flag is set. Therefore the hdr was initially read due
to a predictive prefetch (dmu_zfetch), and has not had a demand (non-prefetch)
read since. But the state is MFU, which implies that it's been accessed
multiple times.
HAS_L2HDR is set, but L2CACHE is not. Therefore we went through arc_read_done()
with the PREFETCH flag set, after it was written to L2ARC.
if (l2arc_noprefetch && HDR_PREFETCH(hdr))
arc_hdr_clear_flags(hdr, ARC_FLAG_L2CACHE);
This doesn't make sense to me - if it was only read via a single prefetch read,
when that read completed, we would clear L2CACHE, so it would never have an
opportunity to be written to L2ARC.
```
> ::stack
arc_buf_remove+0x16(0, fffffe2d3fdfd0b0)
arc_buf_destroy_impl+0x94(fffffe2d3fdfd0b0)
arc_evict_hdr+0xb3(fffffe30c7fd8bb0, ffffffffc00347d0)
arc_evict_state_impl+0x154(fffffe2ce930b468, 2, fffffe320c44a2e0, 0, 53b800)
arc_evict_state+0x12e(ffffffffc0033120, 0, 581800, 2)
arc_adjust_impl+0x44(ffffffffc0033120, 0, 2a05ecaa, 2)
arc_adjust+0x10b()
arc_adjust_cb+0x19(0, fffffe2ce9403e40)
zthr_procedure+0xba(fffffe2ce9403e40)
thread_start+8()
> fffffe2d3fdfd0b0::whatis
fffffe2d3fdfd0b0 is freed from arc_buf_t
> fffffe30c7fd8bb0::print arc_buf_hdr_t
{
b_dva = {
dva_word = [ 0x10, 0x3814e9ab0 ]
}
b_birth = 0x170ab9d
b_type = 2 (ARC_BUFC_METADATA)
b_hash_next = 0
b_flags = 0xf0f0464
(ARC_FLAG_{PREFETCH|PREDICTIVE_PREFETCH|IN_HASH_TABLE|PRIO_ASYNC_READ|BUFC_METADATA|HAS_L1HDR|HAS_L2HDR|COMPRESSED_ARC|COMPRESS_0|COMPRESS_1|COMPRESS_2|
COMPRESS_3})
b_psize = 0x8
b_lsize = 0x20
b_spa = 0x1b440bfb53f69336
b_l2hdr = {
b_dev = 0xfffffe2ce91f0cd0
b_daddr = 0x91237c000
b_l2node = {
list_next = 0xfffffe30c8177160
list_prev = 0xfffffe30d506ad38
}
}
b_l1hdr = {
b_freeze_lock = {
_opaque = [ 0 ]
}
b_freeze_cksum = 0
b_buf = 0xfffffe2d3fdfd0b0
b_bufcnt = 0x1
b_cv = {
_opaque = 0
}
b_byteswap = 0xa
b_state = ARC_mfu
b_arc_node = {
list_next = 0xfffffe2ce9226698
list_prev = 0xfffffe320c44a360
}
b_arc_access = 0x1b586a4f
b_refcnt = {
rc_count = 0
}
b_acb = 0
b_pabd = 0xfffffe30d8ef6c98
}
> fffffe2cf19f1000+3380::blkptr
DVA[0]=<0:7029d356000:2000> [ 0x10, 0x3814e9ab0 ]
DVA[1]=<0:36b82d72000:2000>
DVA[2]=<0:2fae7d96000:2000>
[L0 DNODE] FLETCHER_4 LZ4 LE contiguous unique triple
size=4000L/1000P birth=24161181L/24161181P [0x170ab9d] fill=10
cksum=47c40d0691:f6e8203aa596:1ad111bee454314:f60708759a96e348
> fffffe2cf1be35c0::print dmu_buf_impl_t db.db_data
db.db_data = 0xfffffe2cf19f1000
> fffffe2cf1be35c0::print dmu_buf_impl_t db_objset->os_dsl_dataset
db_objset->os_dsl_dataset = 0
```
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openzfs/openzfs/pull/489#issuecomment-391178074
------------------------------------------
openzfs: openzfs-developer
Permalink:
https://openzfs.topicbox.com/groups/developer/discussions/Te9a3d86d2367f775-Mda4854a01748f41aeb130659
Delivery options: https://openzfs.topicbox.com/groups
