I did a little googling this morning. MS claims that msnbot respects the "pseudo standard*" Crawl-delay directive in robots.txt.
So you might try this in robots.txt User-agent: msnbot Crawl-delay: 20 This should keep msn bot from hitting you more than once every 20 seconds but still get your site indexed if you wish. I just did this and I plan to keep an eye on my logs. * This apparently isn't something which MS dreamt up themselves, several other bots such as Yahoo already supported this. -- TriLUG mailing list : http://www.trilug.org/mailman/listinfo/trilug TriLUG Organizational FAQ : http://trilug.org/faq/ TriLUG Member Services FAQ : http://members.trilug.org/services_faq/ TriLUG PGP Keyring : http://trilug.org/~chrish/trilug.asc
