On 02/17/2010 04:54 PM, Jon Stockill wrote: > Presumably because there are some truly awful bots out there, and google > at least is known to be well behaved.
But the truly awful bots don't look at robots.txt. In fact one of the easiest ways to catch rogue bots is to disallow a small part of the site and then blacklist anybody who goes there. ------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev _______________________________________________ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
