* Platonides <[email protected]> [Wed, 16 Feb 2011 00:40:38 +0100]: > MZMcBride wrote: > > To this end, I filed a JIRA ticket with the Toolserver ops about > getting the > > Wikimedia Bugzilla database replicated with public views: > > https://jira.toolserver.org/browse/TS-901 > > The bugzilla db is on a different machine, it may not be worth it. > > > Leo wrote: > > One thing that bugged me a while: > > Why does the bz robots.txt deny any bots? I think having it indexed by > google & co would'nt exactly make searching harder. > > > > Leo > > A crawler can traverse fast the whole install, which means rendering a > lot of dynamic pages, which bugzilla may not be able to handle. > There is Crawl-delay directive in robots.txt. And one may deny the robots which do not obey it. Dmitriy
_______________________________________________ Wikitech-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikitech-l
