On 02/18/2010 04:07 AM, Rob / EViLSLuT wrote: > Is the syntax of the robots.txt correct? Could be wrong.
Well, technically, it should say "Googlebot" instead of just "Google". But this is such a common mistake that Googlebot answers to the name Google, and no harm is done. > To my knowledge this is what google likes, > > User-agent: * > Disallow: / > > User-agent: Googlebot > Allow: / That's not the recommended form. According to http://www.robotstxt.org/robotstxt.html there is no "Allow:" directive. Certainly there is no advantage to saying "Allow: /" ... and no disadvantage to using the canonical form "Disallow: " which disallows nothing. There are situations where an "Allow:" directive would be helpful, but this is not one of them. Also, due to differences in opinion as to the interpretation of the robots.txt non-standard, it is a bit unpredictable whether bots will respond to the "first match" or "best match" ... so it is good practice to put more-specific directives ahead of less-specific ones. In particular, the "*" wildcard should be last, as it is currently on the site. In any case, the larger point remains: There are plenty of perfectly reasonable, desirable bots that are being excluded by the current file. Conversely there are plenty of truly horrible bots that will never be excluded by any robots.txt file. ------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev _______________________________________________ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
