https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=288777
Bug ID: 288777
Summary: System freezes after upgrading to 14.3-RELEASE
Product: Base System
Version: 14.3-RELEASE
Hardware: amd64
OS: Any
Status: New
Severity: Affects Some People
Priority: ---
Component: kern
Assignee: [email protected]
Reporter: [email protected]
Hello - since upgrading to 14.3-RELEASE, our backup server freezes every 3-4
days. All connections hang (NFS, console, ssh). Have to reboot. I've since
configured watchdogd which will auto-reboot this system now.
dumpdev is set to AUTO but no crash dump is generated. As I understand it, I
would need swap with enough space to handle the amount of memory the system has
+ 10% or so. Right now it's using the default 2GB. I've looked into options but
it looks like the only thing I can do is rebuild this box entirely with more
swap, which I am willing to do if that helps get this fixed.
The only logs I see that might be related (or as a result of this issue) are:
ixl0: ixl_process_adminq: Unable to allocate memory for Admin Queue event!
ixl1: ixl_process_adminq: Unable to allocate memory for Admin Queue event!
watchdog pre-timeout, WD_SOFT_LOG
watchdog pre-timeout, WD_SOFT_PRINTF
panic: watchdog pre-timeout, WD_SOFT_PANIC set
cpuid = 0
time = 1754835209
KDB: stack backtrace:
#0 0xffffffff80ba8f1d at kdb_backtrace+0x5d
#1 0xffffffff80b5aa11 at vpanic+0x161
#2 0xffffffff80b5a8a3 at panic+0x43
#3 0xffffffff809a610b at wd_timeout_cb+0x6b
#4 0xffffffff80b78c79 at softclock_call_cc+0x129
#5 0xffffffff80b7a445 at softclock_thread+0xe5
#6 0xffffffff80b13641 at fork_exit+0x81
#7 0xffffffff81024dee at fork_trampoline+0xe
Uptime: 4d11h35m57s
[reboots]
The Intel adapters are installed but unused.
Hardware:
- Intel(R) Xeon(R) Silver 4214R CPU @ 2.40GHz (2400.00-MHz K8-class CPU)
- 96GB ECC ram
- Intel(R) Ethernet Connection X722 for 10GBASE-T - 2.3.3-k (unused)
- Mellanox MT27800 family adapters (only one in use, internal network)
- 2 x INTEL SSDSC2BB480G7 N2010121 480GB OS
- 24 x WDC WUH721818AL5201 B680 18TB storage
- Swap: 2GB
FreeBSD version:
# freebsd-version -kru
14.3-RELEASE
14.3-RELEASE
14.3-RELEASE-p1
Kernel: GENERIC
Filesystem: ZFS
Workload: This is a backup server. It serves NFS v4 mounts to our internal
network. There is also a remote process that rsyncs data over ssh from ZFS
snapshot dirs. It's very low load.
I'm logging some additional health metrics. This is the last entry before the
last reboot:
last pid: 48943; load averages: 0.47, 0.43, 0.41 up 4+11:34:28 10:12:00
56 processes: 1 running, 55 sleeping
CPU: 0.1% user, 0.0% nice, 0.4% system, 0.0% interrupt, 99.5% idle
Mem: 69M Active, 599M Inact, 848K Laundry, 51G Wired, 56K Buf, 42G Free
ARC: 41G Total, 22G MFU, 11G MRU, 1665K Anon, 1785M Header, 5221M Other
25G Compressed, 32G Uncompressed, 1.27:1 Ratio
Swap: 2048M Total, 485M Used, 1563M Free, 23% Inuse
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
12621 root 1 23 0 52M 37M select 28 86:50 3.66% rsync
ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP XDOM
buf free cache: 432, 0, 11, 0, 11, 0, 0, 0
vm.kmem_map_free: 45821784064
vfs.zfs.arc_max: 0
Device 1K-blocks Used Avail Capacity
/dev/mirror/swap.eli 2097148 496592 1600556 24%
Otherwise, there is no spike in CPU, plenty of memory available. Nothing else
as far as I can see that would be causing this. The server simply freezes and
has to be rebooted.
--
You are receiving this mail because:
You are the assignee for the bug.
