Hi Guys,

Is the syntax of the robots.txt correct? Could be wrong.

To my knowledge this is what google likes,

User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /

Happy flying!
Rob

On 02/18/2010 02:03 AM, John Denker wrote:
> On 02/17/2010 04:54 PM, Jon Stockill wrote:
>
>   
>> Presumably because there are some truly awful bots out there, and google 
>> at least is known to be well behaved.
>>     
> But the truly awful bots don't look at robots.txt.
>
> In fact one of the easiest ways to catch rogue bots
> is to disallow a small part of the site and then
> blacklist anybody who goes there.
>
> ------------------------------------------------------------------------------
> Download Intel® Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs 
> proactively, and fine-tune applications for parallel performance. 
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> _______________________________________________
> Flightgear-devel mailing list
> Flightgear-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/flightgear-devel
>
>   




------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs 
proactively, and fine-tune applications for parallel performance. 
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Flightgear-devel mailing list
Flightgear-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/flightgear-devel

Reply via email to