DOS maybe, certainly looks like it, but I am not convinced. In any event, as soon as I got the IP we blocked it and the problems on the server went away (forgot to mention that part before). The IP is one registered to Comcast and was heading for the hanover website.
sue ----- Original Message ----- From: "Rene Churchill" <[EMAIL PROTECTED]> To: "Vermont Area Group of Unix Enthusiasts" <[email protected]>, [EMAIL PROTECTED] Sent: Saturday, May 10, 2008 10:15:50 PM GMT -05:00 US/Canada Eastern Subject: Re: Apache Problem Hi Sue, Ok, 408 is a timeout error, which is indicative of a DOS attack. The attacker opens a port on your server and just lets the connection sit there until it times out (default 300 seconds). This ties up a connection for those 300 seconds without requiring the webserver to do anything. Max out the connections and nobody can connect to the webserver. So, first thing is to look up that IP address. If it's from someplace relatively untraceable like Romania, then just block it with your firewall or iptables. If the IP is from someplace you think you'd have a chance to complain about, go ahead and report it. You mentioned http://www.thehanovertheatre.org/. Is the IP address for http://www.thehanovertheatre.org/ where the request is coming from or where it is going to? I doubt it is a problem with the code on the page, otherwise you'd be seeing this problem constantly. Next, edit your httpd.conf file and reduce the Timeout value to something much smaller, like 5 or 10 seconds. http://httpd.apache.org/docs/1.3/mod/core.html#timeout An automated attack can still swamp your webserver with a low timeout value, but it will show up in the logfiles much sooner and clean up much faster when you block the IP address. If the problem does continue, let us know and someone can write up a script to monitor the apache logfile for 408 errors and block the naughty IP address w/ iptables. Hope that helps, Rene Sue Fritz wrote: > Sorry I dropped off - yesterday Phil and I went to do some work on our > rack in Waltham. > > I am not 100% sure how much you got from Phil - we just recently started > testing an anti-spoofing filter on our Postini account and now neither > Phil or I are receiving our own posts to this list. I knew this kind of > thing would happen, just didn't realize how much stuff out there > actually validly spoofs. > > So, for the sake of filling in the missing pieces on the apache issue. > We have been having this issue very intermittently for about 4-5 months. > In the past, it would start and go away so fast I couldn't gather any > info. We have a huge volume of logs on this server and I wasn't quite > sure where to start looking without having the opportunity to get an IP > to narrow the search. This time, it lasted long enough for me to do a > netstat and find one IP that had over 500 connections open. In the logs, > that IP generated an equally large number of timeout errors that look > like this.... > > "-" 408 - "-" "-" > > The only web activity reported for that IP is to this site > http://www.thehanovertheatre.org/. > > So at this point I am left with a little more information, but maybe not > enough to really know anything for sure. I might suspect something wrong > with the code on this page - maybe some glitch in the page causes the > java script to blast off a lot of connections??? I am not sure how much > I pursue that angle until I see it happen again in the same way. I did > add code to dump netstats everytime this event occurs - not sure why I > didn't think of that before. > > On the note about the apache settings. I am new to linux, but as a > general philosophy I don't like the "unbounded" approach. In my past > experiences, if one of my webservers croaked, it croaked because someone > put sloppy or untested code on it. We're all human, right? Anything I > can do to mitigate the affects of mistakes is something I want to do. > With that said, I know little about tuning linux/apache, and since most > of Signals stuff works well, I haven't spent a lot of time learning > about this stuff. > > sue > > > ----- Original Message ----- > From: "Rene Churchill" <[EMAIL PROTECTED]> > To: [email protected] > Sent: Thursday, May 8, 2008 7:33:36 PM GMT -05:00 US/Canada Eastern > Subject: Re: Apache Problem > > Sue, Phil, > > We need to see some snippets of the access_log and error_log > logfiles to have a shot at figuring this out. > > As for the config options you've got, those are pretty good for the > vast majority of servers out there. > > MaxRequestsPerChild is there to prevent memory leaks. Since Apache > often load many, many modules written by folks other than the Apache > team, they don't have the highest confidence in their garbage > collection. So it makes sense to restart the process every once in > a while to free up any leaked memory. Don't bother changing it unless > you see the Apache processes chewing up more and more memory over time. > Since your current problem is too many Apache processes, tweaking > MaxRequestsPerChild won't help any. > > Personally, I expect to see the logfiles showing a lot of unusual > HTTP requests, perhaps malformed. Googling the browser string may > be of some use. > > If someone is just opening a connection to port 80 and letting it > timeout, then playing with the TimeOut directive may help. Also > consider disabling KeepAlive. > > http://httpd.apache.org/docs/1.3/mod/core.html#timeout > http://httpd.apache.org/docs/1.3/mod/core.html#keepalive > > Hope that helps, > > Rene > > > Sue Fritz wrote: > > Nick, this might be heading me in the right direction. How do you like > > the looks of these settings from my apache config? > > > > StartServers 10 > > MinSpareServers 5 > > MaxSpareServers 30 > > MaxClients 255 > > xMaxRequestsPerChild 100000 > > > > I am guessing that the MaxRequestsPerChild might be a little high, ya > think? > > > > "The MaxRequestsPerChild directive defines the maximum number of page > > deliveries that each server instance will carry out before closing down > > and respawning. The whole point of a periodic respawn is to prevent > > accumulation of eventual memory leaks. " > > > > > > Sue Fritz > > system admin > > Signal Advertising > > > > ----- Original Message ----- > > From: "Nick Floersch" <[EMAIL PROTECTED]> > > To: [email protected] > > Sent: Thursday, May 8, 2008 2:35:17 PM GMT -05:00 US/Canada Eastern > > Subject: Re: Apache Problem > > > > Phil, this is the Apache config stuff I was thinking of. It *might* be > > helpful only in controlling what your server does with itself when lots > > of connections start rolling in. If The 255 limit is eating up memory > > needed by other servers or processes, you could trim back that limit to > > allow other stuff to use the resources, for example. > > > > Here is an example from my config file: > > --- snip --- > > ## > > ## Server-Pool Size Regulation (MPM specific) > > ## > > > > # prefork MPM > > # StartServers ......... number of server processes to start > > # MinSpareServers ...... minimum number of server processes which are > > kept spare > > # MaxSpareServers ...... maximum number of server processes which are > > kept spare > > # MaxClients ........... maximum number of server processes allowed to > > start > > # MaxRequestsPerChild .. maximum number of requests a server process > > serves > > <IfModule prefork.c> > > StartServers 5 > > MinSpareServers 5 > > MaxSpareServers 10 > > MaxClients 20 > > MaxRequestsPerChild 0 > > </IfModule> > > > > # pthread MPM > > # StartServers ......... initial number of server processes to start > > # MaxClients ........... maximum number of server processes allowed to > > start > > # MinSpareThreads ...... minimum number of worker threads which are > > kept spare > > # MaxSpareThreads ...... maximum number of worker threads which are > > kept spare > > # ThreadsPerChild ...... constant number of worker threads in each > > server process > > # MaxRequestsPerChild .. maximum number of requests a server process > > serves > > <IfModule worker.c> > > StartServers 2 > > MaxClients 150 > > MinSpareThreads 25 > > MaxSpareThreads 75 > > ThreadsPerChild 25 > > MaxRequestsPerChild 0 > > </IfModule> > > > > --- snip --- > > So, those are examples. The docs are at this URL: > > http://httpd.apache.org/docs/2.2/mod/mpm_common.html#serverlimit > > > > But you may also want to look for your timeout directives to decrease > > how long those threads sit waiting. Also, the KeepAlive settings may be > > fouling up somehow, and worth changing a bit. > > --- snip --- > > Timeout 300 > > > > KeepAlive On > > > > MaxKeepAliveRequests 100 > > > > KeepAliveTimeout 15 > > --- snip --- > > > > The URL for that stuff is in the Core directive docs. > > http://httpd.apache.org/docs/2.2/mod/core.html#timeout > > > > http://httpd.apache.org/docs/2.2/mod/core.html#keepalive > > > > http://httpd.apache.org/docs/2.2/mod/core.html#maxkeepaliverequests > > > > http://httpd.apache.org/docs/2.2/mod/core.html#keepalivetimeout > > > > > > Hopefully something here could help tune Apache to handle these requests > > better. I can't help much more than you know on blocking the requests, > > however. I've never seen log entries like yours where there is no IP > > address recorded for the connection. That seems pretty odd to me. > > > > -Nick > > > > -----Original Message----- > > From: Vermont Area Group of Unix Enthusiasts [mailto:[EMAIL PROTECTED] > > On Behalf Of Nick Floersch > > Sent: Thursday, May 08, 2008 1:37 PM > > To: [email protected] > > Subject: Re: Apache Problem > > > > Have you ruled out a deliberate DOS attack? > > > > How quickly does it shoot to 255 processes? There is an option to > > increase the number of child threads the master can split off... but I > > suspect that increasing the limit would just provide more room for it to > > fill up. There might be some way to limit the number of connections per > > client. Still, the Apache worker thread controls might be helpful to > > read about here. > > > > Are all of the connections from the same client IP? Or are you getting > > actions from lots of IPs where the child threads don't quit after > > finishing the transactions? > > > > There is a program called Apache Top, sort of like regular UNIX Top... > > but maybe it would help monitor this stuff? Or is that you are using? > > > > Are these threads initiating processes which interact with a database at > > all? Could the database connections not be closing cleanly, leaving > > Apache thinking it is still reading/writing/connected? > > > > What server side scripting are you using? PHP, Perl, Python, Ruby? > > > > What do the logs say? > > > > -Nick > > > > -----Original Message----- > > From: Vermont Area Group of Unix Enthusiasts [mailto:[EMAIL PROTECTED] > > On Behalf Of Phil Marshall > > Sent: Thursday, May 08, 2008 1:23 PM > > To: [email protected] > > Subject: Apache Problem > > > > Here is a sample from the apache status output. We are getting > > flooded with these. Apache will get filled up to 255 connections with > > these things. This has happened before, but usually it goes away > > quickly. Today it went on for about 45 minutes. Its fine now. Does > > anyone have any idea what this is or how to block it? > > > > 51-9 28349 0/1/1 R 0.00 28 3 0.0 0.02 > > 0.02 ? ? ..reading.. > > 52-9 28350 0/1/1 R 0.00 28 0 0.0 0.000 > > 0.000 ? ? ..reading.. > > 53-9 28351 0/2/2 R 0.00 28 0 0.0 0.000 > > 0.000 ? ? ..reading.. > > 54-9 28352 0/1/1 R 0.00 28 0 0.0 0.000 > > 0.000 ? ? ..reading.. > > 55-9 28354 0/1/1 R 0.00 28 78 0.0 0.00 > > 0.00 ? ? ..reading.. > > 56-9 28355 0/1/1 R 0.00 28 0 0.0 0.000 > > 0.000 ? ? ..reading.. > > > > --------------------- > > Phil Marshall > > Signal Advertising > > 535 Stone Cutters Way > > Montpelier, VT 05602 > > 802-229-4149 > > [EMAIL PROTECTED]
