Re: Dell SAS5 Performance Issue
Matthew Jacob wrote: Is there any news on the performance of this card? I personally have not been able to reproduce the problem. It seems to occur whether in Integrated Raid or not. It seems to be related to specific backplanes and drives. It's an important problem to solve I agree. We have a HP Proliant DL140 g3 that exhibits this (or a somewhat related) problem, to which we can give you remote access (including remote KVM) if that helps? Cheers, Martin ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: netstat fails with memory allocation error and error in kvm_read
Max Laier wrote: On Wednesday 05 January 2005 18:26, J. Martin Petersen wrote: We just had another hard hang, and this time I managed to break to the debugger on the serial console. I got this stack trace http://www.aub.dk/~jmp/fw/trace, which seems to indicate that pf is messing up somehow. This looks like pf.c, rev. 1.25 (HEAD) 1.18.2.6 (RELENG_5) and is an endless loop, not a deadlock. My bad. I hope you can verify that updating to these revisions solves the issue. Please get back to me if not. Thanks in advance. I'm checking sys/contrib/pf out from the releng_5 branch and will compile a new kernel. I'll get back to you if that doesn't help. Thanks for the help Martin ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: netstat fails with memory allocation error and error in kvm_read
I just got another netstat: kvm_read: Bad address. vm.kvm_free was 675278848 two seconds earlier and a couple of seconds later (and also currently). We just had another hard hang, and this time I managed to break to the debugger on the serial console. I got this stack trace http://www.aub.dk/~jmp/fw/trace, which seems to indicate that pf is messing up somehow. I of course didn't have the ddb man or the handbook section on kernel debugging at hand, so I've just printed it for the next time it happens. Both 'show witness' and 'show locks' came up empty. What's the most effective way to hunt down these deadlocks with the online kernel debugger? Best regards Martin ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: netstat fails with memory allocation error and error in kvm_read
You might also log 'sysctl vm.kvm_free' and 'sysctl vm.zone'. I just got another netstat: kvm_read: Bad address. vm.kvm_free was 675278848 two seconds earlier and a couple of seconds later (and also currently). The output from vm.zone a couple of seconds after: vm.zone: ITEMSIZE LIMIT USEDFREE REQUESTS pfosfp: 28,0,155,226, 1085 pfospfen:108,0,293, 31, 2051 pfiaddrpl:92,0, 5, 79, 28 pfstatescrub: 8,0, 26676, 28540, 37487180 pffrcent: 12,50141, 0, 0,0 pffrcache:48,10062, 0, 0,0 pffrag: 48,0, 0,234, 378044 pffrent: 16, 5075, 0,203, 934804 pfrkentry: 152,0, 4, 48,8 pfrktable: 248,0, 4, 28, 12 pfpooladdrpl: 68,0, 21, 91, 126 pfaltqpl:128,0, 19, 71, 133 pfstatepl: 256,65010, 14396, 23014, 29835551 pfrulepl:560,0,248,221, 1693 pfsrctrpl:80,0, 0, 0,0 FFS2 dinode: 256,0, 32828,427, 1154768 FFS1 dinode: 128,0, 0, 0,0 FFS inode: 140,0, 32828,352, 1154768 SWAPMETA:276,63672, 9, 47, 40 rtentry: 132,0,219,187, unpcb: 140,17220, 9, 47, 5217 ripcb: 180, 8140, 0, 44, 15 sackhole: 16,0, 0,203, 3107 tcpreass: 20, 1183, 0,169, 134 hostcache:88,15400, 3, 85, 39 syncache:108,15372, 0, 72, 374 tcptw:56, 1675, 2,132, 48 tcpcb: 448, 8136, 4, 23, 489 inpcb: 180, 8140, 6, 38, 489 udpcb: 180, 8140, 11, 33, 6297398 socket: 324, 8136, 24, 36, 6303121 KNOTE:68,0, 0,112, 12538429 PIPE:384,0, 2, 78, 251051 DIRHASH:1024,0, 1621, 11, 4318 NAMEI: 1024,0, 3, 17, 131064562 L VFS Cache: 291,0, 38,456, 8482 S VFS Cache: 68,0, 29954, 5550, 1214821 VNODEPOLL:64,0, 0,118,2 VNODE: 264,0, 33205, 20,33205 ata_request: 200,0, 0, 38, 2322599 g_bio: 132,0, 0, 2204, 4301499 MbufClust: 2048,17216,906,124, 12121216 Mbuf:256,0,907,188, 2520398988 Packet: 256,0,740,355, 2728311990 VMSPACE: 300,0, 23, 81, 726569 UPCALL: 44,0, 0, 0,0 KSEGRP: 104,0,144, 31, 144 TID: 140,0, 1, 53,1 THREAD: 388,0,144, 6, 144 PROC:452,0, 73, 71, 726627 Files:68,0, 71,153, 122894112 4096: 4096,0,129, 60, 746477 2048: 2048,0,130, 30,24039 1024: 1024,0, 23, 61, 14341894 512: 512,0,244, 92, 656520 256: 256,0,256, 1094, 7153814 128: 128,0, 1842,318, 15428344 64: 64,0, 4481, 1183, 8923514 32: 32,0, 2595,682, 408850 16: 16,0, 1930,303, 22115093 DP fakepg:72,0, 0, 0,0 PV ENTRY: 24, 944530, 5540, 17080, 315728548 MAP ENTRY:68,0,439, 1241, 32683625 KMAP ENTRY: 68,32200, 40,464, 557 MAP: 192,0, 7, 33,5 VM OBJECT: 132,0, 32560, 1428, 14688659 128 Bucket: 524,0,966, 0,0 64 Bucket: 268,0, 30, 40,0 32 Bucket: 140,0, 30, 26,0 16 Bucket:76,0, 15, 35,0 UMA Hash:128,0, 2, 28,0 UMA RCntSlab:104,0,515, 3,0 UMA Slabs:64,0,793, 33,0 UMA Zones:88,0, 73, 7,0 UMA Kegs:136,0, 73, 23,0 vm.zone a couple of seconds before: ITEMSIZE LIMIT USEDFREE REQUESTS pfosfp: 28,0,155,226, 1085 pfospfen:108,0,293, 31, 2051 pfiaddrpl:92,0, 5, 79, 28 pfstatescrub: 8,0, 27146, 28070, 37485813 pffrcent: 12,50141, 0, 0,
RE: netstat fails with memory allocation error and error in kvm_read
We are trying to gather some debug information for a problem that is difficult to diagnose, as the machine always ends up hard frozen (does not do anything, can not break to debugger, does not respond to keyboard, etc.), so we are dumping output from netstat, vmstat, iostat etc. quite often. The cron jobs fail ever so often with error messages I do not quite understand, and I can not find anything relevant in the archives. Can anyone shed some light on this? Command:netstat -r Error message: netstat: kvm_read: Bad address Debug before: http://www.aub.dk/~jmp/fw/debug-2004.12.09-21.34.41.gz Debug after: http://www.aub.dk/~jmp/fw/debug-2004.12.09-21.35.06.gz # errors: 7 Command:netstat -an Error message:netstat: sysctl: net.inet.udp.pcblist: Cannot allocate memory Debug before: http://www.aub.dk/~jmp/fw/debug-2004.12.09-07.38.48.gz Debug after: http://www.aub.dk/~jmp/fw/debug-2004.12.09-07.39.04.gz # errors: 3 You appear to be running out of kernel memory. Since you're capturing the output of vmstat -m, you should check that for any bins that are growing at a high rate of speed. Seems possible that its in pf :) I've checked the numbers from just before the freeze (it's within 15 secs) with two sets of data: From a fresh boot and five minutes minutes before the freeze. Here are the stuff that changed significantly between the fresh boot and just before the freeze: Just before the freeze (http://www.aub.dk/~jmp/fw/tmp/debug-2004.12.11-22.59.01.gz): AR driver 2 1K268K 2922822 64,256,512,2048 kqueue 0 0K 38K 13304405 128,1024 UFS dirhash 44488K107K 2559 16,32,64,128,256,512,1024,2048,4096 freeblks13 4K 29K 103030 256 freefrag 0 0K 1K 164217 32 allocindir 28718K162K 1966413 64 indirdep 1 1K209K 9925 32 allocdirect27 4K 18K 301048 128 inodedep18 131K150K 164032 128,256 routetbl 56647K 67K 800649 16,32,64,128,256 subproc99 301K849K 1873146 32,4096 Five minutes before the freeze (http://www.aub.dk/~jmp/fw/tmp/debug-2004.12.11-22.55.42.gz): AR driver 2 1K268K 2921793 64,256,512,2048 kqueue 0 0K 38K 13296556 128,1024 UFS dirhash 44488K107K 2559 16,32,64,128,256,512,1024,2048,4096 freeblks 1 1K 29K 102978 256 freefrag 0 0K 1K 164153 32 allocindir 0 0K162K 1965284 64 indirdep 0 0K209K 9921 32 allocdirect 1 1K 18K 300886 128 inodedep14 130K150K 163954 128,256 routetbl 56246K 67K 800255 16,32,64,128,256 subproc99 301K849K 1872250 32,4096 From a fresh boot (http://www.aub.dk/~jmp/fw/tmp/debug-2004.12.11-23.31.31.gz): AR driver 2 1K190K23450 64,256,512,2048 kqueue 0 0K 3K 1062 128,1024 UFS dirhash3613K 13K 42 16,32,512,2048,4096 freeblks 11529K 29K 253 256 freefrag 0 0K 1K 51 32 allocindir 2 1K135K 3332 64 indirdep10 1K173K 630 32 allocdirect 2 1K 40K 456 128 inodedep 137 145K168K 506 128,256 routetbl 30626K 27K 495 16,32,64,128,256 subproc 107 317K466K 1554 32,4096 The numbers for pflog and pf_if does not change at all. I checked vmstat -z, and the highest pf-related entries we're actually decreasing at the time of the deadlock, but I noticed the following: VM OBJECT: 132,0, 31508, 2132, 14364021 128 Bucket: 524,0,727, 1,0 64 Bucket: 268,0, 23, 19,0 32 Bucket: 140,0, 34, 22,0 16 Bucket:76,0, 15, 35,0 Can you or anyone else deduce anything from the numbers? If not, I'll whip something together that runs vmstat -m ever so often, parses the output and remove the non-increasing entries so it'll be easier to spot the trends. Thanks, Martin ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
netstat fails with memory allocation error and error in kvm_read
Hi We are trying to gather some debug information for a problem that is difficult to diagnose, as the machine always ends up hard frozen (does not do anything, can not break to debugger, does not respond to keyboard, etc.), so we are dumping output from netstat, vmstat, iostat etc. quite often. The cron jobs fail ever so often with error messages I do not quite understand, and I can not find anything relevant in the archives. Can anyone shed some light on this? Command:netstat -r Error message: netstat: kvm_read: Bad address Debug before: http://www.aub.dk/~jmp/fw/debug-2004.12.09-21.34.41.gz Debug after:http://www.aub.dk/~jmp/fw/debug-2004.12.09-21.35.06.gz # errors: 7 Command:netstat -an Error message: netstat: sysctl: net.inet.udp.pcblist: Cannot allocate memory Debug before: http://www.aub.dk/~jmp/fw/debug-2004.12.09-07.38.48.gz Debug after:http://www.aub.dk/~jmp/fw/debug-2004.12.09-07.39.04.gz # errors: 3 Command:netstat -an Error message: netstat: sysctl: net.inet.raw.pcblist: Cannot allocate memory Debug before: http://www.aub.dk/~jmp/fw/debug-2004.12.10-22.19.50.gz Debug after:http://www.aub.dk/~jmp/fw/debug-2004.12.10-22.20.10.gz # errors: 6 The number of errors are in the last five days. Kernelconfig (FIREWALL), dmesg, pf.conf etc. can be found at http://www.aub.dk/~jmp/fw/ The commands we are running are: netstat -i, netstat -m, netstat -s, netstat -an, netstat -r vmstat, vmstat -i, vmstat -f, vmstat -m, vmstat -z iostat ps waux every 15 seconds. We are running the same commands (plus pfctl -vvsa) from root's cron every minute. The hard lockups seems to be connected to our use of pf and altq, that is at least the problem we're trying to gather debug information for. Thanks in advance Martin ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
