On Mon, 2008-07-14 at 14:22 +0200, Henning Brauer wrote: > perfect analysis! > > looks like the only sane thing to do in that case is to bail and not > send the icmp.
I've compiled a new kernel with the patch. The machine is no longer crashing on pf_send_icmp(). However, I now see memory leaking until the machine locks up (it doesn't crash but its network becomes unusable). Unfortunately, it then also puts all CARP interfaces in MASTER state, though the other node works perfectly as master already. This will, of course, knock down our entire network until I manually put down the carp interfaces. I have increased kern.maxclusters to gain more time for debugging of the memory leak. However, all I could find out so far is that lots of mbufs are allocated while there is no significant traffic to be handled (remember the machine is the CARP backup). The machine crashes within 15 minutes after reboot. Because of the line wrapping in this email, I've also put the output of netstat and vmstat online) http://www.ini.uzh.ch/~stephan/vmstat+netstat.txt # vmstat -m Memory statistics by bucket size Size In Use Free Requests HighWater Couldfree 16 3549 10275 304244 1280 7725 32 303 209 51063 640 0 64 2968 360 93244 320 89 128 511 65 5665 160 0 256 189 131 12817 80 1065 512 351 9 3326 40 0 1024 2313 11 3302 20 0 2048 33 1 1536 10 0 4096 28 1 6834 5 0 8192 12 0 12 5 0 16384 6 0 6 5 0 32768 5 0 5 5 0 65536 1 0 1 5 0 Memory usage type by bucket size Size Type(s) 16 devbuf, pcb, routetbl, ifaddr, sysctl, UFS mount, dirhash, in_multi, exec, xform_data, VM swap, UVM amap, UVM aobj, USB, USB device, packet tags, temp 32 devbuf, pcb, routetbl, ifaddr, UFS mount, sem, dirhash, proc, VFS cluster, in_multi, ether_multi, xform_data, VM swap, UVM amap, USB, temp, AGP Memory 64 devbuf, pcb, routetbl, ifaddr, vnodes, sem, dirhash, ip_moptions, in_multi, pfkey data, UVM amap, USB, NDP, temp 128 devbuf, routetbl, ifaddr, vnodes, ttys, exec, UVM amap, USB, USB device, NDP, temp, AGP Memory 256 devbuf, routetbl, ifaddr, sysctl, ioctlops, vnodes, shm, VM map, proc, NFS srvsock, NFS daemon, newblk, UVM amap, USB, USB device, temp 512 devbuf, pcb, ifaddr, ioctlops, mount, UFS mount, shm, dirhash, ttys, exec, UVM amap, USB device, temp 1024 devbuf, ioctlops, namecache, proc, ttys, exec, UVM amap, UVM aobj, crypto data, temp 2048 devbuf, ifaddr, ioctlops, UFS mount, pagedep, VM swap, UVM amap, temp 4096 devbuf, ioctlops, UFS mount, MSDOSFS mount, memdesc, temp 8192 devbuf, NFS node, namecache, UFS quota, UFS mount, ISOFS mount, inodedep 16384 devbuf, namecache, UVM amap 32768 devbuf, VM swap 65536 VM swap Memory statistics by type Type Kern Type InUse MemUse HighUse Limit Requests Limit Limit Size(s) devbuf 3808 2545K 2545K 39322K 3880 0 0 16,32,64,128,256,512,1024,2048,4096,8192,16384,32768 pcb 30 4K 4K 39322K 78 0 0 16,32,64,512 routetbl 280 27K 44K 39322K 1400 0 0 16,32,64,128,256 ifaddr 143 25K 25K 39322K 145 0 0 16,32,64,128,256,512,2048 sysctl 2 1K 1K 39322K 2 0 0 16,256 ioctlops 0 0K 4K 39322K 5457 0 0 256,512,1024,2048,4096 mount 4 2K 2K 39322K 4 0 0 512 NFS node 1 8K 8K 39322K 1 0 0 8192 vnodes 1256 83K 87K 39322K 1312 0 0 64,128,256 namecache 3 25K 25K 39322K 3 0 0 1024,8192,16384 UFS quota 1 8K 8K 39322K 1 0 0 8192 UFS mount 17 35K 35K 39322K 17 0 0 16,32,512,2048,4096,8192 shm 2 1K 1K 39322K 2 0 0 256,512 VM map 4 1K 1K 39322K 4 0 0 256 sem 2 1K 1K 39322K 2 0 0 32,64 dirhash 30 6K 6K 39322K 30 0 0 16,32,64,512 proc 15 3K 3K 39322K 15 0 0 32,256,1024 VFS cluster 0 0K 1K 39322K 26 0 0 32 NFS srvsock 1 1K 1K 39322K 1 0 0 256 NFS daemon 1 1K 1K 39322K 1 0 0 256 ip_moptions 5 1K 1K 39322K 5 0 0 64 in_multi 123 5K 5K 39322K 124 0 0 16,32,64 ether_multi 64 2K 3K 39322K 65 0 0 32 ISOFS mount 1 8K 8K 39322K 1 0 0 8192 MSDOSFS mount 1 4K 4K 39322K 1 0 0 4096 ttys 420 263K 263K 39322K 420 0 0 128,512,1024 exec 0 0K 2K 39322K 3090 0 0 16,128,512,1024 pfkey data 1 1K 1K 39322K 2 0 0 64 xform_data 0 0K 1K 39322K 18 0 0 16,32 pagedep 1 2K 2K 39322K 1 0 0 2048 inodedep 1 8K 8K 39322K 1 0 0 8192 newblk 1 1K 1K 39322K 1 0 0 256 VM swap 7 75K 75K 39322K 7 0 0 16,32,2048,32768,65536 UVM amap 3819 233K 349K 39322K 349090 0 0 16,32,64,128,256,512,1024,2048,16384 UVM aobj 2 2K 2K 39322K 2 0 0 16,1024 USB 74 7K 7K 39322K 74 0 0 16,32,64,128,256 USB device 21 9K 9K 39322K 21 0 0 16,128,256,512 memdesc 1 4K 4K 39322K 1 0 0 4096 crypto data 1 1K 1K 39322K 1 0 0 1024 packet tags 0 0K 1K 39322K 8 0 0 16 NDP 24 3K 3K 39322K 28 0 0 64,128 temp 112 14K 18K 39322K 116753 0 0 16,32,64,128,256,512,1024,2048,4096 AGP Memory 2 1K 1K 39322K 2 0 0 32,128 Memory Totals: In Use Free Requests 3405K 252K 482097 Memory resource pool statistics Name Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle extentpl 20 248 0 197 1 0 1 1 0 8 0 phpool 32 36864 0 0 291 0 291 291 0 8 0 pmappl 84 2469 0 2443 2 0 2 2 0 8 1 vmsppl 188 2469 0 2443 4 0 4 4 0 8 2 vmmpepl 88 207056 0 205462 106 0 106 106 0 179 71 vmmpekpl 88 59313 0 59192 3 0 3 3 0 8 0 aobjpl 52 1 0 0 1 0 1 1 0 8 0 amappl 44 113985 0 112738 50 0 50 50 0 45 30 anonpl 16 161598 0 156491 34 0 34 34 0 31 11 bufpl 124 2817 0 123 85 0 85 85 0 8 0 mbpl 256 618808 0 553629 4075 0 4075 4075 1 4096 1 mclpl 2048 185575 0 120403 32591 0 32591 32591 4 32768 4 sockpl 212 1944 0 1906 3 0 3 3 0 8 0 procpl 344 2486 0 2443 9 0 9 9 0 8 4 processpl 20 2486 0 2443 1 0 1 1 0 8 0 zombiepl 72 2443 0 2443 1 0 1 1 0 8 1 ucredpl 80 848 0 836 1 0 1 1 0 8 0 pgrppl 24 1448 0 1424 1 0 1 1 0 8 0 sessionpl 48 33 0 10 1 0 1 1 0 8 0 pcredpl 24 2486 0 2443 1 0 1 1 0 8 0 lockfpl 52 12 0 10 1 0 1 1 0 8 0 filepl 88 15335 0 15245 4 0 4 4 0 8 2 fdescpl 296 2487 0 2443 8 0 8 8 0 8 4 pipepl 72 1246 0 1242 2 0 2 2 0 8 1 kqueuepl 192 3 0 0 1 0 1 1 0 8 0 knotepl 64 9 0 0 1 0 1 1 0 8 0 sigapl 316 2469 0 2443 7 0 7 7 0 8 4 wqtasks 20 227 0 227 1 0 1 1 0 8 1 wdcspl 96 3416 0 3416 1 0 1 1 0 8 1 scxspl 132 3 0 3 1 0 1 1 0 8 1 namei 1024 26059 0 26059 2 0 2 2 0 8 2 vnodes 148 1582 0 0 59 0 59 59 0 8 0 nchpl 72 1636 0 57 29 0 29 29 0 8 0 ffsino 184 1664 0 91 72 0 72 72 0 8 0 dino1pl 128 1664 0 91 51 0 51 51 0 8 0 dirhash 1024 37 0 0 10 0 10 10 0 128 0 pfrulepl 824 442 0 10 111 0 111 111 0 8 2 pfstatepl 204 23449 0 21322 173 0 173 173 0 264 30 pfstatekeypl 108 23449 0 21374 86 5 81 86 0 8 8 pfpooladdrpl 68 27 0 0 1 0 1 1 0 8 0 pfrktable 1240 144 0 72 48 0 48 48 0 334 0 pfrkentry 156 1089 0 0 42 0 42 42 0 13462 0 pfosfpen 108 1392 0 696 30 11 19 19 0 8 0 pfosfp 28 814 0 407 3 0 3 3 0 8 0 rtentpl 116 74 0 4 2 0 2 2 0 8 0 tcpcbpl 400 190 0 179 3 0 3 3 0 8 1 tcpqepl 16 6 0 6 1 0 1 1 0 13 1 synpl 184 185 0 185 1 0 1 1 0 8 1 plimitpl 152 133 0 122 1 0 1 1 0 8 0 inpcbpl 216 1875 0 1858 2 0 2 2 0 8 1 # netstat -m In use 150665K, total allocated 154056K; utilization 97.8% 65183 mbufs in use: 65178 mbufs allocated to data 1 mbuf allocated to packet headers 4 mbufs allocated to socket names and addresses 65178/65186/65536 mbuf clusters in use (current/peak/max) 146676 Kbytes allocated to network (14% in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines
