>
>> I'd monitor this over time and see if the rise is sudden or gradual.
>> And whether you can correlate with log entries. Is it happening after
>> the LCP timeout? Is the LCP timeout happening after this has already
>> risen?
>>
>> e.g.
>>
>> $ while true; do netstat -m | grep mbuf.2048; sleep 5; done | ts
>>
>> Capture of the output of 'systat mbuf' might also give a clue (it updates
>> frequently, you could leave it running in ssh).
>
> I have set up some capture around this and I will report back once
> it fails again and hopefully that will give some additional hints.
I don’t think we need to wait until it fails over again, looking at the
current rate that the number of mbufs is increasing it is roughly
50 per minute, this puts it on a collision course with the 2-3 day
LCP timeouts. So perhaps the LCP timeout is actually being caused by
the kernel running out of space for any more allocations.
Is there some way to dig deeper into what is creating and not releasing
these?
# snapshot of recent netstat -m | grep mbuf.2048
Jan 13 14:08:01 23123/23200 mbuf 2048 byte clusters in use (current/peak)
Jan 13 14:09:02 23168/23256 mbuf 2048 byte clusters in use (current/peak)
Jan 13 14:10:02 23205/23288 mbuf 2048 byte clusters in use (current/peak)
Jan 13 14:11:01 23255/23344 mbuf 2048 byte clusters in use (current/peak)
Jan 13 14:12:01 23439/23520 mbuf 2048 byte clusters in use (current/peak)
Jan 13 14:13:01 23484/23576 mbuf 2048 byte clusters in use (current/peak)
Jan 13 14:14:01 23539/23632 mbuf 2048 byte clusters in use (current/peak)
Jan 13 14:15:01 23578/23648 mbuf 2048 byte clusters in use (current/peak)
Jan 13 14:16:01 23632/23720 mbuf 2048 byte clusters in use (current/peak)
Jan 13 14:17:01 23668/23768 mbuf 2048 byte clusters in use (current/peak)
Jan 13 14:18:01 23706/23784 mbuf 2048 byte clusters in use (current/peak)
Jan 13 14:19:01 23843/23936 mbuf 2048 byte clusters in use (current/peak)
Jan 13 14:20:01 23882/23976 mbuf 2048 byte clusters in use (current/peak)
Jan 13 14:21:01 23927/24016 mbuf 2048 byte clusters in use (current/peak)
Jan 13 14:22:01 23985/24088 mbuf 2048 byte clusters in use (current/peak)
Jan 13 14:23:01 24047/24120 mbuf 2048 byte clusters in use (current/peak)
Jan 13 14:24:01 24077/24160 mbuf 2048 byte clusters in use (current/peak)
Jan 13 14:25:01 24135/24224 mbuf 2048 byte clusters in use (current/peak)
Jan 13 14:26:01 24277/24384 mbuf 2048 byte clusters in use (current/peak)
Jan 13 14:27:01 24320/24400 mbuf 2048 byte clusters in use (current/peak)
Jan 13 14:28:01 24399/24480 mbuf 2048 byte clusters in use (current/peak)
Jan 13 14:29:01 24439/24528 mbuf 2048 byte clusters in use (current/peak)
Jan 13 14:30:01 24494/24576 mbuf 2048 byte clusters in use (current/peak)
Jan 13 14:31:01 24644/24728 mbuf 2048 byte clusters in use (current/peak)
Jan 13 14:32:01 24746/24840 mbuf 2048 byte clusters in use (current/peak)
Jan 13 14:33:01 24806/24896 mbuf 2048 byte clusters in use (current/peak)
# systat mbuf
IFACE RING LIVELOCKS SIZE ALIVE LWM HWM CWM
System mbufs 0 256 30554 1917
mcl2k 2048 24948 3127
mcl2k2 2112 524 46
mcl4k 4096 0 8
mcl8k 8192 0 7
mcl9k 9216 0 1
mcl12k 12288 0 4
mcl16k 16384 0 3
mcl64k 65536 0 4
lo0
igc0 0 2048 39 11 1023 39
1 2048 11 11 1023 11
2 2048 11 11 1023 11
3 2048 11 11 1023 11
igc1 0 2048 91 11 1023 91
1 2048 88 11 1023 88
2 2048 85 11 1023 85
3 2048 89 11 1023 89
aq0 2048 5 5 2047 5
2048 5 5 2047 5
2048 5 5 2047 5
2048 5 5 2047 5
2048 5 5 2047 5
2048 5 5 2047 5
2048 5 5 2047 5
2048 5 5 2047 5
aq1 2048 7 5 2047 7
2048 7 5 2047 7
2048 8 5 2047 8
2048 7 5 2047 7
2048 8 5 2047 8
2048 7 5 2047 7
2048 7 5 2047 7
2048 7 5 2047 7
enc0
pppoe0
veb0
vport0
wg0
pflog0
