On December 1, 2005 10:29 PM I wrote: > Subject: [Axiom-developer] RE: [EMAIL PROTECTED]: > [RHH#FKL-86268-984]: move Axiom to shared dedicated server] > > ... > I have also cut-back the robots.txt to be more selective > about what we allow the search engines to do. > ...
After thinking about this a little more I realized that one of the things that causes a higher access rate for the robot search engines and spiders is that we now have multiple virtual urls for the same MathAction web sites: http://page.axiom-developer.org (wiki and portal) http://www.axiom-developer.org (wiki and portal) http://wiki.axiom-developer.org http://portal.axiom-developer.org Since the robots can not tell which of these urls map to which web sites, it is a result it is possible that we are being scanned 3 times for the same content! So today, I made url specific urls that block all robots from the first two urls but allow full indexing of both the wiki and the portal (except for bibliography and SandBox pages): User-agent: * Disallow: /SandBox Disallow: /bibliography Disallow: /refs/axiombib Disallow: /refs/redbib ------- I also changed http://www.axiom-developer.org and http://axiom-developer.org to map directly to http://wiki.axiom-developer.org So far, this seems to have resulted in significantly lower rate of access by these robots. As a result, the performance of the MathAction websites now seems to me to be very fast. When you have some time, please try the Axiom wiki and portal sites and comment on how fast (or slow) they seem to you. Your comments on these changes are welcome. Regards, Bill Page. _______________________________________________ Axiom-developer mailing list [email protected] http://lists.nongnu.org/mailman/listinfo/axiom-developer
