Re: Help with high LA
On Thu, 12 Feb 2009 10:04:41 -0200 (BRST) sc...@centroin.com.br wrote: I need help for some strange problem with one of my servers, that can cost my job. It's a FreeBSD 7.0-RELEASE-p5/amd64 running on a Dell PowerEdge III as a Virtual machine of VMware ESXi. There are only two VM in this box, and one of them (basicly a mail server) is running fine. The problem is with high loads on the other one, that runs (besides other services) http and pop3. TOP show LA from 40 to 90 most of the time. I thought, at first, that was a disk botleneck due to some big mailboxes, or something related to some Apache (2.2.9) fine tuning, but it's something else. If I stop pop3 and apache services (the most active of the box), the LA drops to 1~2. Starting only one of them (any one) the LA rise to 20~40. Sugesting that it's not tied to a specific service. I did a test running just pop3 (Qpopper), pointing the mail spool to a empty directory, to make shure that it's not a disk problem. And the LA also goes to sky (~30). The same happens with only apache running pointing to a simple http page. The console shows messages like: ipfw: install_state: Too many dynamic rules net.inet.ip.fw.dyn_max: 4096 # (here) Maximum number of dynamic rules. When you hit this limit, no more dynamic rules can be installed until old ones expire. To see which traffic is creating 'too many' dynamic rules, check: # ipfw -ted show | less -S ++G (-td for just active rules, but the expired ones tell useful stories) I know I must review my rules and limit the number of keep-state entries, but a tryed to rise the number of dynamic buckets via sysctl: sysctl -w net.inet.ip.fw.dyn_buckets=2048 But it seems it's not working, since the number of current buckets doesn't pass 256: net.inet.ip.fw.curr_dyn_buckets: 256 But did you remember to flush? :) See ipfw(8) under 'SYSCTL VARIABLES'. You might also want to monitor and/or play with some of the other net.inet.ip.fw.dyn_* sysctls to see what's happening and how many dynamic rules you need with comfortable headroom for your workload/s. For TCP, keepalive and *lifetime timeouts may be relevant. I tend to use stateful rules for outbound UDP, and stateless setup and established rules for TCP services here, but your needs may differ. I tryed to make some OS tuning, from the handbook, like increase the maxcon: kern.ipc.somaxconn: 2048 but nothing seems to work. Other entries in the logs: Feb 12 09:06:20 host1 inetd[1248]: accept (for ftp): Software caused connection abort Feb 12 09:06:20 host1 inetd[1248]: accept (for pop3): Software caused connection abort I need some clues to undestand what is happening. Thank you, - Marcelo Yes, 'Too many dynamic rules'; further connections will surely fail. cheers, Ian ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Help with high LA
Hi All, I need help for some strange problem with one of my servers, that can cost my job. It's a FreeBSD 7.0-RELEASE-p5/amd64 running on a Dell PowerEdge III as a Virtual machine of VMware ESXi. There are only two VM in this box, and one of them (basicly a mail server) is running fine. The problem is with high loads on the other one, that runs (besides other services) http and pop3. TOP show LA from 40 to 90 most of the time. I thought, at first, that was a disk botleneck due to some big mailboxes, or something related to some Apache (2.2.9) fine tuning, but it's something else. If I stop pop3 and apache services (the most active of the box), the LA drops to 1~2. Starting only one of them (any one) the LA rise to 20~40. Sugesting that it's not tied to a specific service. I did a test running just pop3 (Qpopper), pointing the mail spool to a empty directory, to make shure that it's not a disk problem. And the LA also goes to sky (~30). The same happens with only apache running pointing to a simple http page. The console shows messages like: ipfw: install_state: Too many dynamic rules I know I must review my rules and limit the number of keep-state entries, but a tryed to rise the number of dynamic buckets via sysctl: sysctl -w net.inet.ip.fw.dyn_buckets=2048 But it seems it's not working, since the number of current buckets doesn't pass 256: net.inet.ip.fw.curr_dyn_buckets: 256 I tryed to make some OS tuning, from the handbook, like increase the maxcon: kern.ipc.somaxconn: 2048 but nothing seems to work. Other entries in the logs: Feb 12 09:06:20 host1 inetd[1248]: accept (for ftp): Software caused connection abort Feb 12 09:06:20 host1 inetd[1248]: accept (for pop3): Software caused connection abort I need some clues to undestand what is happening. Thank you, - Marcelo ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Help with high LA
sc...@centroin.com.br wrote: Hi All, I need help for some strange problem with one of my servers, that can cost my job. It's a FreeBSD 7.0-RELEASE-p5/amd64 running on a Dell PowerEdge III as a Virtual machine of VMware ESXi. There are only two VM in this box, and one of them (basicly a mail server) is running fine. The problem is with high loads on the other one, that runs (besides other services) http and pop3. TOP show LA from 40 to 90 most of the time. I thought, at first, that was a disk botleneck due to some big mailboxes, or something related to some Apache (2.2.9) fine tuning, but it's something else. If I stop pop3 and apache services (the most active of the box), the LA drops to 1~2. Starting only one of them (any one) the LA rise to 20~40. Sugesting that it's not tied to a specific service. I did a test running just pop3 (Qpopper), pointing the mail spool to a empty directory, to make shure that it's not a disk problem. And the LA also goes to sky (~30). The same happens with only apache running pointing to a simple http page. The console shows messages like: ipfw: install_state: Too many dynamic rules I know I must review my rules and limit the number of keep-state entries, but a tryed to rise the number of dynamic buckets via sysctl: sysctl -w net.inet.ip.fw.dyn_buckets=2048 But it seems it's not working, since the number of current buckets doesn't pass 256: net.inet.ip.fw.curr_dyn_buckets: 256 I tryed to make some OS tuning, from the handbook, like increase the maxcon: kern.ipc.somaxconn: 2048 but nothing seems to work. Other entries in the logs: Feb 12 09:06:20 host1 inetd[1248]: accept (for ftp): Software caused connection abort Feb 12 09:06:20 host1 inetd[1248]: accept (for pop3): Software caused connection abort I need some clues to undestand what is happening. Thank you, - Marcelo Me, I would get rid of inetd and just run the services as daemons. Since these are services which always need to be up there is no need for inetd. I also usually don't run firewalls on my service servers, but rather locate them in a subnet where there is a dedicated box for firewalling. I don't have the experience with your type of VM configuration, but I have the feeling that you could push the firewall function somewhere else. Dump inetd and if it is acceptable (e.g. you are behind something else) try running without ipfw. You probably need to do some in depth profiling of your problem box, and I'm probably not at the level of expertise you need. There are others in the list which can be more helpful. Just thought I'd toss out what I'd look at first if it me. -Mike ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
