Tim, On December 1, 2005 3:17 AM you wrote: > > Axiom-developer has been moved from a shared-virtual server > (basically vmware-ish) to a shared dedicated server. the > shared dedicated server is a dual-processor hyper-threaded > machine with only 4 customers so we basically get a full > processor and a gig of memory to ourselves. > > Hopefully this fixes the problem. Let me know. >
Thanks. So far this is great! The machine is actual a quad-processor and even a single threaded operation (using only one processor) seems to be at least twice as fast on the old server. With a quad processor, there is one more thing that I can do to improve performance - it is possible to run more than one zope instance, accessing the same zope database. This has some advantages over just allowing zope to run more threads because of a limitation in the way threads are handled in Python. It might take me a day or so to setup an 2nd zope instance, so I will wait on that for now and just see how it goes. Some yesterday when I was having a lot of trouble with the old server I changed the robots.txt file to block all search engines and spiders. User-agent: * Disallow: / These robots have become particularly aggressive lately - you just have to look at the web statistics to see where a lot of the cpu cycles where going. http://page.axiom-developer.org/usage Recently their access rate was frequently overrunning the server's capacity. With the modified robots.txt that was slowed down a lot (but not stopped since there are some robots out there that do not play nicely). This morning when I had a spare moment and after reading your email about the new server, I temporarily opened up access to the search robots again. I was surprised to see them all suddenly "jump" onto the new server and it wasn't long before they had overrun the zope process again! So even with this machine is still have to be a little careful. Adding another zope instance would help prevent this. Today I did a couple of quick things that might have improved the zope performance a little: 1) I did a re-pack of the zope database. Normally this is done once a week on Monday morning around 00:05 AM. 2) I re-built the catalogue indexes for both the wiki and the portal. This second one might have been more beneficial than I originally thought. I have also cut-back the robots.txt to be more selective about what we allow the search engines to do. User-agent: MSNbot Disallow: / User-agent: * Disallow: /SandBox Disallow: /zope/mathaction/SandBox Disallow: /bibliography Disallow: /zope/Plone/bibliography Disallow: /refs/axiombib Disallow: /zope/Plone/refs/axiombib Disallow: /refs/redbib Disallow: /zope/Plone/refs/redbib -------- Some explanations about the contents of robots.txt: I did not see much point in allowing the Microsoft Network MSNbot in (which is one of the robots that has been quite aggressive on our site) since we are already be aggressively scanned by Googlebot. The /SandBox pages are supposed to be just drafts anyway so I don't think we want this indexed externally in any case. The /bibliography pages involve a lot of cpu processing each time they are accessed and I have not found anyway to make this more efficient yet, so for now I have blocked robot access to them. Right now this seems enough keep things in check but as time goes on, I might have to add more items to the disallow list. (It's a bit awkward that the robots.txt standard only allows Disallow commands and not Allow commands ... ) If anyone has any suggestions about how to optimize the robots.txt file, I would be very glad to receive them. Finally, Tim. I am still concerned about the fact that you (personally) seem to be bearing all the cost of running the axiom-developer.org server. I expect that this new configuration is going to cost significantly more since it is dedicated server. Let's talk about what we can do to make this more equitable. Thanks again for setting this up. Regards, Bill Page. > > ------- Start of forwarded message ------- > Date: Wed, 30 Nov 2005 23:26:03 -0600 > From: "RH Helpdesk" <[EMAIL PROTECTED]> > Subject: [RHH #FKL-86268-984]: move Axiom to shared dedicated server > Reply-To: "RH Helpdesk" <[EMAIL PROTECTED]> > > > This has been completed. > > Admin > RoseHosting.com > > > On Wed, 30 Nov 2005 18:01:11 -0600, [EMAIL PROTECTED] wrote: > > IP: 72.23.23.52 > > > > Please move the axiom-developer.org > > domain (209.135.140.38) from a shared > > virtual server to a shared dedicated > > server and send me mail when the > > process completes. > > > > Tim Daly > > Axiom Lead Developer > > - ------- End of forwarded message ------- > > _______________________________________________ Axiom-developer mailing list [email protected] http://lists.nongnu.org/mailman/listinfo/axiom-developer
