>> > My apache web server has been very slow lately and webalizer charts >> > show page accesses at 5x normal with other stats normal. I'm thinking >> > scrapers? How do you guys deal with this? Do you identify the IP >> > (how?) and ban it (how?)? >> > >> > - Grant >> >> I used netstat to identify the IP and I see that I can use it with >> "deny from" in httpd.conf. It seems to be over now, but this type of >> thing happens periodically. How can I be alerted to this type of >> situation when it starts so I can block the IP right away? > > You will need to configure quotas probably using something like: > > http://www.howtoforge.com/mod_cband_apache2_bandwidth_quota_throttling > > Not sure if it is possible to differentiate between rogue and legit clients, > other than by checking your logs to see what was blocked.
Turns out it was a "legit" bot. Watch out for this one: Mozilla/5.0 (compatible; discobot/1.0; +http://discoveryengine.com/discobot.html) It's bad that a single IP can bring down my http isn't it? - Grant
