(I sent this to -lpsg and -steer in a single message last night but
it doesn't appear to have made it to steer, so here it is again.
Sorry if this winds up being a dupe.)
Per our discussion at LPSG, I've had some fun with rewrite rules and
seem to have gotten the load on the server down significantly.
During the meeting I added a quick Webalizer report to our web
server, which can be found here:
http://sparkplug.kernel-panic.org/stats/
That report lets us see how the majority of traffic is reaching us.
The answer is Google searches, and if you're easily offended I
suggest you skip the section of the report showing what people are
searching to find us. :)
Although referrals from Google.com are top on the list, referrals
from other foreign Google.<country> sites make up quite a chunk.
I've added rules to make Apache deny anyone coming to us via those
foreign Googles. My list is not complete, but I used the top 20 list
on Webalizer to guide me. If someone has a text processing itch and
wants to parse the complete list of official Google sites (http://
www.google.com/supported_domains) into the following format:
RewriteCond %{HTTP_REFERER} ^http://www.google.pl [OR]
RewriteCond %{HTTP_REFERER} ^http://www.google.com.au [OR]
I'd be more than happy to paste the entire list into our config.
I also made explicit rules that deny visits to some of the high
traffic Member sites, such as 'allaboutx'.
With these rules in place, I've been able to re-enable the Member
sites for everyone else and the server load is staying low.
Tracy, you may not have to do the CacheFu/Squid thing, but you're
more than welcome to if you want. One other thought - our Apache
server already has mod_proxy installed, giving it similar
functionality to Squid... is there a way to integrate that with
CacheFu instead? That would save the installation of yet another
daemon that takes memory.
That's all for now, heading to bed. I'll get into the Plone specific
problems later.
--
Joshua Penix http://www.binarytribe.com
Binary Tribe Linux Integration Services & Network Consulting
--
[email protected]
http://www.kernel-panic.org/cgi-bin/mailman/listinfo/kplug-steer
