Oh great and wise FreeBSD gurus,
I've been running FreeBSD boxes for about five years with great results (up to 6 at the moment), but recently one of my machines has started to seriously act up. Every time a heavy disk operation (say, tar'ing a 1 gig directory) occurs the system slows to a crawl, and requests to apache/php/mysql sites hosted on it just hang.
The system is a dual p3 1.13ghz box with a gig of ram and mirrored 80 gig WD800BB drives on a Promise TX2 controller. The raid isn't degraded. There's a dedicated 1.5 gig swap partition and a swap file on the /usr partition. We had some apache processes go nuts one time, which is why I added the swap file.
We run about 15 jails on the machine, with MySQL in the server proper and apache/php running inside the jails. I initially thought it was a rogue process taking down the machine, but it seems to be that any heavy disk activity for more than a few minutes brings about the slowdown. It doesn't happen instantly, but after a minute or two things will slow to a crawl.
I've recompiled the kernel a few times, upgraded to the latest 4-STABLE rev, and even turned on device polling, but nothing seems to be helping. It doesn't seem to happen on another machine we have with identical hardware.
My sysctl.conf:
kern.ipc.somaxconn=4096 net.inet.tcp.sendspace=32768 net.inet.tcp.recvspace=32768 net.inet.icmp.drop_redirect=1 net.inet.icmp.log_redirect=1 net.inet.ip.redirect=0 net.inet6.ip6.redirect=0 net.link.ether.inet.max_age=1200 net.inet.icmp.bmcastecho=0 net.inet.icmp.maskrepl=0 kern.maxfiles=65536 kern.ipc.shm_use_phys=1 kern.polling.enable=1
And a netstat -m:
301/928/131072 mbufs in use (current/peak/max):
301 mbufs allocated to data
287/874/32768 mbuf clusters in use (current/peak/max)
1980 Kbytes allocated to network (2% of mb_map in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routinesAnd here's a typical systat -v snapshot while the machine's 'ok':
3 users Load 0.32 0.38 0.31 Aug 6 00:03
Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER
Tot Share Tot Share Free in out in out
Act 221588 38656 747652 117796 39404 count 4 3
All 1024156 41620 1546136 144132 pages 18 5
Interrupts
Proc:r p d s w Csw Trp Sys Int Sof Flt 21 cow 1156 total
2 2 70 343 63322119 1156 57 397 186992 wire fxp0 irq2
623848 act 13 ohci0 irq9
4.4%Sys 1.0%Intr 2.5%User 0.0%Nice 92.1%Idl 176096 inact 11 mux irq10
| | | | | | | | | | 37220 cache fdc0 irq6
==+> 2184 free 1004 clk irq0
daefr 128 rtc irq8
Namei Name-cache Dir-cache 15 prcfr
Calls hits % hits % 5 react
126 125 99 pdwake
340 zfod pdpgs
Disks ad4 ad6 fd0 md0 119 ofod 1 intrn
KB/t 0.00 16.72 0.00 0.00 34 %slo-z 114304 buf
tps 0 11 0 0 401 tfree 173 dirtybuf
MB/s 0.00 0.17 0.00 0.00 70310 desiredvnodes
% busy 0 9 0 0 64089 numvnodes
54829 freevnodes
And here's a systat -v snapshop while the machine's choking:
4 users Load 0.39 0.35 0.31 Aug 6 00:08
Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER
Tot Share Tot Share Free in out in out
Act 191344 34248 728736 117268 51916 count 1 6
All 1024676 37500 2075520 144188 pages 2 67
Interrupts
Proc:r p d s w Csw Trp Sys Int Sof Flt 29 cow 1698 total
5 2 70 573 74423171 1699 225 367 180904 wire fxp0 irq2
640404 act 335 ohci0 irq9
5.7%Sys 1.9%Intr 7.5%User 0.0%Nice 84.9%Idl 153116 inact 236 mux irq10
| | | | | | | | | | 50252 cache fdc0 irq6
===+>>>> 1664 free 999 clk irq0
daefr 128 rtc irq8
Namei Name-cache Dir-cache 93 prcfr
Calls hits % hits % 1 react
8693 8196 94 12 0 pdwake
308 zfod 2693 pdpgs
Disks ad4 ad6 fd0 md0 135 ofod intrn
KB/t 98.81 16.61 0.00 0.00 43 %slo-z 114304 buf
tps 13 225 0 0 1277 tfree 278 dirtybuf
MB/s 1.23 3.64 0.00 0.00 70310 desiredvnodes
% busy 2 99 0 0 64089 numvnodes
52125 freevnodes
Thoughts? Is there any way to force a machine to limit the monopolization of a disk controller by a process?
--
Jeff Kramer [EMAIL PROTECTED] http://www.keika.org/ _______________________________________________ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
