On Aug 7, 2007, at 3:34 PM, Scott Hill wrote:
OK, I have a minute so I was wondering what is robots.txt and what is it's purpose and/or uses?
The robots.txt file communicates to complying search engines which pages should not be indexed, and which search engines should not crawl a site.
For example: http://www.apple.com/robots.txt http://discussions.apple.com/robots.txt http://php.net/robots.txt For more info: http://www.robotstxt.org/ _______________________________________________ UPHPU mailing list [email protected] http://uphpu.org/mailman/listinfo/uphpu IRC: #uphpu on irc.freenode.net
