Jon Jensen wrote:
It's a way for web sites to tell bots/spiders/crawlers how to behave
(e.g. what they can and cannot view), though it is of course entirely
up to the bot to comply.
http://www.robotstxt.org/wc/norobots.html
We find that there are a ton of bots that do the exact opposite of what
you tell them to. I'd use .htaccess to deny all traffic in directories
and files that no one needs to access via http, to password protect
admin / private directories, and to just deny bots in directories you
simply don't want indexed.
Robots.txt files are great though for telling google which pictures it
can and can't use :-)
-V
_______________________________________________
UPHPU mailing list
[email protected]
http://uphpu.org/mailman/listinfo/uphpu
IRC: #uphpu on irc.freenode.net
