Sue, Phil,
We need to see some snippets of the access_log and error_log
logfiles to have a shot at figuring this out.
As for the config options you've got, those are pretty good for the
vast majority of servers out there.
MaxRequestsPerChild is there to prevent memory leaks. Since Apache
often load many, many modules written by folks other than the Apache
team, they don't have the highest confidence in their garbage
collection. So it makes sense to restart the process every once in
a while to free up any leaked memory. Don't bother changing it unless
you see the Apache processes chewing up more and more memory over time.
Since your current problem is too many Apache processes, tweaking
MaxRequestsPerChild won't help any.
Personally, I expect to see the logfiles showing a lot of unusual
HTTP requests, perhaps malformed. Googling the browser string may
be of some use.
If someone is just opening a connection to port 80 and letting it
timeout, then playing with the TimeOut directive may help. Also
consider disabling KeepAlive.
http://httpd.apache.org/docs/1.3/mod/core.html#timeout
http://httpd.apache.org/docs/1.3/mod/core.html#keepalive
Hope that helps,
Rene
Sue Fritz wrote:
Nick, this might be heading me in the right direction. How do you like
the looks of these settings from my apache config?
StartServers 10
MinSpareServers 5
MaxSpareServers 30
MaxClients 255
MaxRequestsPerChild 100000
I am guessing that the MaxRequestsPerChild might be a little high, ya think?
"The MaxRequestsPerChild directive defines the maximum number of page
deliveries that each server instance will carry out before closing down
and respawning. The whole point of a periodic respawn is to prevent
accumulation of eventual memory leaks. "
Sue Fritz
system admin
Signal Advertising
----- Original Message -----
From: "Nick Floersch" <[EMAIL PROTECTED]>
To: [email protected]
Sent: Thursday, May 8, 2008 2:35:17 PM GMT -05:00 US/Canada Eastern
Subject: Re: Apache Problem
Phil, this is the Apache config stuff I was thinking of. It *might* be
helpful only in controlling what your server does with itself when lots
of connections start rolling in. If The 255 limit is eating up memory
needed by other servers or processes, you could trim back that limit to
allow other stuff to use the resources, for example.
Here is an example from my config file:
--- snip ---
##
## Server-Pool Size Regulation (MPM specific)
##
# prefork MPM
# StartServers ......... number of server processes to start
# MinSpareServers ...... minimum number of server processes which are
kept spare
# MaxSpareServers ...... maximum number of server processes which are
kept spare
# MaxClients ........... maximum number of server processes allowed to
start
# MaxRequestsPerChild .. maximum number of requests a server process
serves
<IfModule prefork.c>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 20
MaxRequestsPerChild 0
</IfModule>
# pthread MPM
# StartServers ......... initial number of server processes to start
# MaxClients ........... maximum number of server processes allowed to
start
# MinSpareThreads ...... minimum number of worker threads which are
kept spare
# MaxSpareThreads ...... maximum number of worker threads which are
kept spare
# ThreadsPerChild ...... constant number of worker threads in each
server process
# MaxRequestsPerChild .. maximum number of requests a server process
serves
<IfModule worker.c>
StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>
--- snip ---
So, those are examples. The docs are at this URL:
http://httpd.apache.org/docs/2.2/mod/mpm_common.html#serverlimit
But you may also want to look for your timeout directives to decrease
how long those threads sit waiting. Also, the KeepAlive settings may be
fouling up somehow, and worth changing a bit.
--- snip ---
Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15
--- snip ---
The URL for that stuff is in the Core directive docs.
http://httpd.apache.org/docs/2.2/mod/core.html#timeout
http://httpd.apache.org/docs/2.2/mod/core.html#keepalive
http://httpd.apache.org/docs/2.2/mod/core.html#maxkeepaliverequests
http://httpd.apache.org/docs/2.2/mod/core.html#keepalivetimeout
Hopefully something here could help tune Apache to handle these requests
better. I can't help much more than you know on blocking the requests,
however. I've never seen log entries like yours where there is no IP
address recorded for the connection. That seems pretty odd to me.
-Nick
-----Original Message-----
From: Vermont Area Group of Unix Enthusiasts [mailto:[EMAIL PROTECTED]
On Behalf Of Nick Floersch
Sent: Thursday, May 08, 2008 1:37 PM
To: [email protected]
Subject: Re: Apache Problem
Have you ruled out a deliberate DOS attack?
How quickly does it shoot to 255 processes? There is an option to
increase the number of child threads the master can split off... but I
suspect that increasing the limit would just provide more room for it to
fill up. There might be some way to limit the number of connections per
client. Still, the Apache worker thread controls might be helpful to
read about here.
Are all of the connections from the same client IP? Or are you getting
actions from lots of IPs where the child threads don't quit after
finishing the transactions?
There is a program called Apache Top, sort of like regular UNIX Top...
but maybe it would help monitor this stuff? Or is that you are using?
Are these threads initiating processes which interact with a database at
all? Could the database connections not be closing cleanly, leaving
Apache thinking it is still reading/writing/connected?
What server side scripting are you using? PHP, Perl, Python, Ruby?
What do the logs say?
-Nick
-----Original Message-----
From: Vermont Area Group of Unix Enthusiasts [mailto:[EMAIL PROTECTED]
On Behalf Of Phil Marshall
Sent: Thursday, May 08, 2008 1:23 PM
To: [email protected]
Subject: Apache Problem
Here is a sample from the apache status output. We are getting
flooded with these. Apache will get filled up to 255 connections with
these things. This has happened before, but usually it goes away
quickly. Today it went on for about 45 minutes. Its fine now. Does
anyone have any idea what this is or how to block it?
51-9 28349 0/1/1 R 0.00 28 3 0.0 0.02
0.02 ? ? ..reading..
52-9 28350 0/1/1 R 0.00 28 0 0.0 0.000
0.000 ? ? ..reading..
53-9 28351 0/2/2 R 0.00 28 0 0.0 0.000
0.000 ? ? ..reading..
54-9 28352 0/1/1 R 0.00 28 0 0.0 0.000
0.000 ? ? ..reading..
55-9 28354 0/1/1 R 0.00 28 78 0.0 0.00
0.00 ? ? ..reading..
56-9 28355 0/1/1 R 0.00 28 0 0.0 0.000
0.000 ? ? ..reading..
---------------------
Phil Marshall
Signal Advertising
535 Stone Cutters Way
Montpelier, VT 05602
802-229-4149
[EMAIL PROTECTED]
