Seems to happen very random.
I have two systems which do this. Both are 2-node CARP.
It is only kernel which is -STABLE on those two. Userland is from a snap
approx. 14 days old.
Update to stable kernel is done recently, because of those crashes.
Kernel before is running OK on more heavily loaded machines.
Here is a trace from a crash just 50min ego.
I can’t do any thing directly in gdb (ddb.panic=1) as this is IPMI console and
kbd not working.
(gdb) file /var/crash/bsd.0
Reading symbols from /var/crash/bsd.0...(no debugging symbols found)...done.
(gdb) target kvm /var/crash/bsd.0.core
#0 0xffffffff8131c7d4 in dumpsys ()
(gdb) where
#0 0xffffffff8131c7d4 in dumpsys ()
#1 0x0000000380000000 in ?? ()
#2 0x00000003024c0000 in ?? ()
#3 0x0000000000000005 in ?? ()
#4 0xffffffff8135e680 in chopen ()
#5 0xffffffff8131c90b in dumpsys ()
#6 0x0000000000000021 in ?? ()
#7 0xffff80002127be68 in ?? ()
#8 0xffffffff81328434 in amd_cpu_cacheinfo ()
#9 0xffffffff81328340 in x86_print_cacheinfo ()
#10 0xffff80002127be68 in ?? ()
#11 0xffffffff811958d6 in srp_enter ()
#12 0xffffffff811958d6 in srp_enter ()
#13 0xffffffff811a4986 in addlog ()
#14 0x0000003000000008 in ?? ()
#15 0xffff80002127bf78 in ?? ()
#16 0xffff80002127be98 in ?? ()
#17 0x757ff7a94f468953 in ?? ()
#18 0x0000000000000000 in ?? ()
(gdb)
[prdsrv0003.sto1.se]-[21:42:31]# ps -N /var/crash/bsd.0 -M
/var/crash/bsd.0.core -O paddr
PID PADDR TT STAT TIME COMMAND
25250 ffffff0401574dc0 ?? Is+ 0:00.00 (ksh)
12560 ffffff0401574280 ?? Is+ 0:00.01 (getty)
30357 ffffff04015744c0 ?? Is+ 0:00.01 (getty)
12471 ffffff0401574700 ?? Is+ 0:00.01 (getty)
8797 ffffff0401574940 ?? Is+ 0:00.01 (getty)
21030 ffffff0401574b80 ?? Is+ 0:00.01 (getty)
[prdsrv0003.sto1.se]-[21:45:12]# vmstat -N /var/crash/bsd.0 -M
/var/crash/bsd.0.core -m
Memory statistics by bucket size
Size In Use Free Requests HighWater Couldfree
16 3320 264 22393 1280 0
32 1226 54 4254 640 0
64 2992 80 6582 320 0
128 23616 1056 65855 160 292
256 339 29 4879 80 0
512 284 12 930 40 0
1024 318 6 1111 20 36
2048 51 1027 1618 10 757
4096 36 2 442 5 0
8192 17 8 47 5 2
16384 4 0 6 5 0
32768 6 0 9 5 0
65536 3 0 543 5 0
131072 3 0 3 5 0
262144 1 0 1 5 0
524288 2 0 2 5 0
Memory usage type by bucket size
Size Type(s)
16 devbuf, pcb, routetbl, UFS mount, dirhash, ACPI, ip_moptions, exec,
VM swap, UVM amap, UVM aobj, USB, USB device, temp
32 devbuf, pcb, routetbl, ifaddr, UFS mount, sem, dirhash, ACPI,
in_multi, exec, UVM amap, USB, USB device, temp
64 devbuf, routetbl, ifaddr, vnodes, UFS mount, dirhash, ACPI, proc,
in_multi, ether_multi, VM swap, UVM amap, USB, NDP, temp
128 devbuf, pcb, routetbl, ifaddr, sem, dirhash, ACPI, NFS srvsock,
ip_moptions, in_multi, ttys, pfkey data, UVM amap, USB, USB device,
temp
256 devbuf, routetbl, ifaddr, ioctlops, iov, vnodes, UFS mount, shm,
VM map, ACPI, exec, xform_data, UVM amap, USB, USB device, NDP, temp
512 devbuf, ifaddr, ioctlops, iov, UFS mount, dirhash, ACPI, file desc,
ttys, xform_data, newblk, UVM amap, USB device, temp
1024 devbuf, pcb, ioctlops, iov, mount, shm, ACPI, file desc, proc, ttys,
exec, UVM amap, USB device, crypto data, temp
2048 devbuf, pcb, ioctlops, UFS mount, ACPI, VM swap, UVM amap, UVM aobj,
temp
4096 devbuf, pcb, ifaddr, ioctlops, UFS mount, proc, UVM amap, USB,
memdesc, temp
8192 devbuf, pcb, ttys, pagedep, UVM amap, USB, temp
16384 devbuf, pcb, NFS daemon, MSDOSFS mount, temp
32768 devbuf, pcb, UFS quota, UFS mount, ISOFS mount, inodedep
65536 devbuf, pcb, temp
131072 devbuf, pcb
262144 devbuf
524288 VM swap
Memory statistics by type Type Kern
Type InUse MemUse HighUse Limit Requests Limit Limit Size(s)
devbuf 1326 838K 3078K 78644K 6142 0 0
16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536,131072,262144
pcb 119 276K 404K 78644K 177 0 0
16,32,128,1024,2048,4096,8192,16384,32768,65536,131072
routetbl 232 11K 12K 78644K 999 0 0 16,32,64,128,256
ifaddr 116 29K 29K 78644K 119 0 0
32,64,128,256,512,4096
ioctlops 0 0K 4K 78644K 1508 0 0
256,512,1024,2048,4096
iov 0 0K 1K 78644K 118 0 0 256,512,1024
mount 9 9K 9K 78644K 9 0 0 1024
vnodes 1159 73K 73K 78644K 1168 0 0 64,256
UFS quota 1 32K 32K 78644K 1 0 0 32768
UFS mount 37 80K 80K 78644K 37 0 0
16,32,64,256,512,2048,4096,32768
shm 2 2K 2K 78644K 2 0 0 256,1024
VM map 2 1K 1K 78644K 2 0 0 256
sem 2 1K 1K 78644K 2 0 0 32,128
dirhash 24 5K 5K 78644K 27 0 0 16,32,64,128,512
ACPI 24996 2937K 3048K 78644K 67490 0 0
16,32,64,128,256,512,1024,2048
file desc 18 13K 13K 78644K 19 0 0 512,1024
proc 13 10K 10K 78644K 13 0 0 64,1024,4096
NFS srvsock 1 1K 1K 78644K 1 0 0 128
NFS daemon 1 16K 16K 78644K 1 0 0 16384
ip_moptions 6 1K 1K 78644K 6 0 0 16,128
in_multi 24 2K 2K 78644K 25 0 0 32,64,128
ether_multi 25 2K 2K 78644K 26 0 0 64
ISOFS mount 1 32K 32K 78644K 1 0 0 32768
MSDOSFS mount 1 16K 16K 78644K 1 0 0 16384
ttys 420 308K 308K 78644K 420 0 0
128,512,1024,8192
exec 0 0K 3K 78644K 559 0 0 16,32,256,1024
pfkey data 1 1K 1K 78644K 2 0 0 128
xform_data 0 0K 1K 78644K 4 0 0 256,512
pagedep 1 8K 8K 78644K 1 0 0 8192
inodedep 1 32K 32K 78644K 1 0 0 32768
newblk 1 1K 1K 78644K 1 0 0 512
VM swap 7 1171K 1171K 78644K 7 0 0
16,64,2048,524288
UVM amap 3365 204K 215K 78644K 26469 0 0
16,32,64,128,256,512,1024,2048,4096,8192
UVM aobj 2 3K 3K 78644K 2 0 0 16,2048
USB 67 39K 41K 78644K 69 0 0
16,32,64,128,256,4096,8192
USB device 24 3K 3K 78644K 24 0 0
16,32,128,256,512,1024
memdesc 1 4K 4K 78644K 1 0 0 4096
crypto data 1 1K 1K 78644K 1 0 0 1024
NDP 17 2K 2K 78644K 17 0 0 64,256
temp 195 77K 205K 78644K 3203 0 0
16,32,64,128,256,512,1024,2048,4096,8192,16384,65536
Memory Totals: In Use Free Requests
6224K 2289K 108675
Memory resource pool statistics
Name Size Requests Fail InUse Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
pfsync 72 569 0 0 105 104 1 1 0 8 1
arp 56 38 0 10 1 0 1 1 0 8 0
inpcbpl 288 762352 0 5633 2784 2288 496 837 0 8 13
plimitpl 152 34 0 25 1 0 1 1 0 8 0
syncache 248 380713 0 0 85 84 1 37 0 8 1
sackhl 24 8 0 0 8 8 0 1 0 8 0
tcpqe 32 382335 0 0 1 0 1 1 0 8 1
tcpcb 560 762248 0 5609 5668 4635 1033 1670 0 8 17
rtmask 32 13 0 13 1 0 1 1 0 8 0
rtentry 192 203 0 100 7 0 7 7 0 8 0
strpolpl 48 2 0 0 2 2 0 1 0 8 0
strprocpl 2448 4 0 0 4 4 0 2 0 8 0
pfosfp 40 840 0 420 5 0 5 5 0 8 0
pfosfpen 112 1420 0 710 21 0 21 21 0 8 0
pfstitem 24 765582 0 22683 359 209 150 182 0 8 0
pfstkey 104 766147 0 22683 2106 1454 652 795 0 8 0
pfstate 312 766147 0 23183 9042 6976 2066 2544 0 8 13
pfrule 1336 28 0 12 10 5 5 10 0 8 0
dirhash 1024 35 0 25 7 0 7 7 0 8 0
dino1pl 128 1714 0 1661 54 0 54 54 0 8 0
ffsino 240 1714 0 1661 98 0 98 98 0 8 0
nchpl 144 2846 0 2734 102 0 102 102 0 8 0
uvmvnodes 72 1731 0 1731 32 0 32 32 0 8 0
vnodes 200 1731 0 1731 92 0 92 92 0 8 0
namei 1024 7073 0 0 62 62 0 3 0 8 0
ehcixfer 264 91 0 4 1 0 1 1 0 8 0
xhcixfer 240 85 0 2 1 0 1 1 0 8 0
scxspl 192 39727 0 1 44 43 1 2 0 8 0
sigapl 432 342 0 37 6 1 5 5 0 8 0
knotepl 112 5699826 0 205 11 2 9 10 0 8 0
kqueuepl 320 21 0 21 2 0 2 2 0 8 0
pipepl 120 162 0 2 5 4 1 1 0 8 0
fdescpl 440 343 0 38 6 1 5 5 0 8 0
filepl 120 765638 0 204 44 37 7 10 0 8 0
lockfpl 88 3 0 1 1 0 1 1 0 8 0
sessionpl 64 24 0 19 1 0 1 1 0 8 0
pgrppl 40 36 0 23 1 0 1 1 0 8 0
ucredpl 96 106 0 31 1 0 1 1 0 8 0
zombiepl 144 305 0 0 8 8 0 1 0 8 0
processpl 656 365 0 60 12 1 11 11 0 8 0
procpl 576 365 0 60 10 1 9 10 0 8 0
sosppl 88 756322 0 5543 801 641 160 215 0 8 6
sockpl 392 762500 0 5737 4060 3351 709 1182 0 8 10
mcl2k 2048 5867578 0 18 1835 1768 67 247 4 6144 58
mbufpl 256 16854833 0 5680 1800 1293 507 736 1 768 37
bufpl 256 38744 0 1656 104 0 104 104 0 8 0
anonpl 16 35236 0 6769 31 3 28 29 0 1014 0
amappl 72 25862 0 3293 66 4 62 64 0 75 0
dma4096 4096 1 0 0 1 1 0 1 0 8 0
dma512 512 11 0 1 1 0 1 1 0 8 0
dma256 256 10 0 0 1 1 0 1 0 8 0
dma64 64 11 0 0 1 1 0 1 0 8 0
dma32 32 9 0 0 1 1 0 1 0 8 0
dma16 16 29 0 0 1 1 0 1 0 8 0
aobjpl 64 1 0 1 1 0 1 1 0 8 0
uaddrrnd 24 343 0 38 1 0 1 1 0 8 0
uaddrbest 32 2 0 2 1 0 1 1 0 8 0
uaddr 24 343 0 38 1 0 1 1 0 8 0
vmmpekpl 168 74439 0 4 1 0 1 1 0 8 0
vmmpepl 168 47712 0 5612 281 35 246 261 0 357 0
vmsppl 240 342 0 37 3 0 3 3 0 8 0
pdppl 4096 342 0 37 47 10 37 42 0 8 0
pvpl 32 674839 0 15266 142 15 127 133 0 265 0
pmappl 168 342 0 37 2 0 2 2 0 8 0
extentpl 40 149 0 53 1 0 1 1 0 8 0
phpool 112 15838 0 3433 121 0 121 121 0 8 0
In use 23134K, total allocated 29244K; utilization 79.1%
> On 17 sep. 2015, at 11:20, Stuart Henderson <s...@spacehopper.org> wrote:
>
> On 2015-09-17, mxb <m...@alumni.chalmers.se> wrote:
>> Hey,
>> getting panics with 5.8-STABLE kernel.
>>
>> panic: mix_enter: locking against myself
>> Starting stack trace…
>> panic() at panic+0x10b
>> mtx_enter() at mtx_enter+0x60
>> sofree() at sofree+0xa0
>> in_pcbdetach() at in_pcbdetach+0x40
>> tcp_close() at tcp_close+0xad
>> tcp_timer_2msl() at tcp_timer_2msl+0x90
>> softclock() at softclock+0x315
>> softintr_dispatch() at softintr_dispatch+0x8b
>> Xsoftclock() at Xsoftclock+0x1f
>> ——————interrupt———————————
>> (null)() at 0x8
>> end of kernel
>> end trace frame: 0x112000000000001, count: 247
>> end of stack trace
>>
>>
>
> A bit more information about when this is happening might be useful..
