On Fri, May 21, 2004 at 03:55:57PM +0200, Robert Brenstein wrote: > > > >Not a virus, just microsoft releasing a beta version of their spider on > >the web. We had it bring a site to a crawl. It's hitting a java-based > >app that created a new java session every time msnbot hit it. Fortunately > >the resources freed themselves quickly, but having a hundred instances > >running was not good. I added msnbot to the robots.txt file and it > >stopped spidering the site within a few minutes. Once the spider is out > >of beta I'll remove them from robots.txt. > > > > How did you identify it in the robots.txt file ?
User-agent: msnbot Disallow: / -ken +------------------------------------------------------------------------ | TO UNSUBSCRIBE from this list: | http://lists.isite.net/listgate/analog-help/unsubscribe.html | | Digest version: http://lists.isite.net/listgate/analog-help-digest/ | Usenet version: news://news.gmane.org/gmane.comp.web.analog.general | List archives: http://www.analog.cx/docs/mailing.html#listarchives +------------------------------------------------------------------------
