OK, I have a minute so I was wondering what is robots.txt and what is it's purpose and/or uses?
It's a way for web sites to tell bots/spiders/crawlers how to behave (e.g. what they can and cannot view), though it is of course entirely up to the bot to comply.
http://www.robotstxt.org/wc/norobots.html _______________________________________________ UPHPU mailing list [email protected] http://uphpu.org/mailman/listinfo/uphpu IRC: #uphpu on irc.freenode.net
