On Friday 04 February 2011 13:38:14 Joe Btfsplk wrote: > No ideas yet on what "automated software that doesn't follow /robots.txt > is forbidden," means?
robots.txt is a file put on some websites as a directive to robots. If you run a wiki, and you want only current versions, not the hundreds of previous versions of every page, indexed, you could put a directive in robots.txt, or label the pages themselves as "noindex nofollow". Automated software that ignores such directives is likely to eat up huge amounts of bandwidth and create copies that are many times bigger than the original. cmeclax *********************************************************************** To unsubscribe, send an e-mail to majord...@torproject.org with unsubscribe or-talk in the body. http://archives.seul.org/or/talk/