Thanks for your comments. I've changed the robots.txt to allow all bots that actually respect the file in anticipation of the server switch.
On Thu, Feb 18, 2010 at 7:32 AM, John Denker <[email protected]> wrote: > On 02/18/2010 04:07 AM, Rob / EViLSLuT wrote: > >> Is the syntax of the robots.txt correct? Could be wrong. > > Well, technically, it should say "Googlebot" instead of just > "Google". But this is such a common mistake that Googlebot > answers to the name Google, and no harm is done. > >> To my knowledge this is what google likes, >> >> User-agent: * >> Disallow: / >> >> User-agent: Googlebot >> Allow: / > > That's not the recommended form. According to > http://www.robotstxt.org/robotstxt.html > > there is no "Allow:" directive. Certainly there is no advantage > to saying "Allow: /" ... and no disadvantage to using the canonical > form "Disallow: " which disallows nothing. > > There are situations where an "Allow:" directive would be helpful, > but this is not one of them. > > Also, due to differences in opinion as to the interpretation of > the robots.txt non-standard, it is a bit unpredictable whether > bots will respond to the "first match" or "best match" ... so > it is good practice to put more-specific directives ahead of > less-specific ones. In particular, the "*" wildcard should be > last, as it is currently on the site. > > In any case, the larger point remains: There are plenty of > perfectly reasonable, desirable bots that are being excluded by > the current file. Conversely there are plenty of truly horrible > bots that will never be excluded by any robots.txt file. > > -- Simon Hollier ------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev _______________________________________________ Flightgear-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/flightgear-devel
