Thanks for the information. I documented it. https://savannah.gnu.org/maintenance/MailSystem
-- Sylvain On Mon, Jan 29, 2007 at 11:29:07AM -0500, Joshua Ginsberg via RT wrote: > If my institutional memory serves me correctly, we disallowed robots on > lists.gnu.org because the frequency and intensity with which the spiders > crawled lists made mail and web service on the box slow to a crawl -- > the disks were hammered and began to fry from all of the I/O. We made > the decision that, since our archives are available and googleable from > the numerous mirrors out there, given the choice between faster mail > service and search-engine indexed archives, we'd prefer the former. > > Lists is still under a relatively constant load. I'm attaching the load > average graph for lists for the last 24 hours, for your viewing pleasure. > > If there is not a googleable service out there that fits your needs, > we'll look into caching mechanisms and distributed load to try to bring > the spiders back to lists. Otherwise, this request may have to wait > until we migrate lists to the 4-core Opteron machine. > > -jag > > > [beuc - Thu Jan 25 11:13:41 2007]: > > > > Hi, > > > > Search engines do not index lists.gnu.org (and lists.nongnu.org) at > > all, probably due to the current robots.txt: > > > > User-agent: * > > Disallow: / > > > > This is blocking when I try to search across several mailing lists. > > > > Can you allow indexing of public archives? > > > > Thanks, > > > > > > > -- > Joshua Ginsberg <[EMAIL PROTECTED]> > Free Software Foundation - Senior Systems Administrator >
