John -
robots.txt is a standard tool, as you said, that directs search engine robots
away from specified directories and files. I use this at work to ensure, for
instance, that maintenance areas of our website will not be crawled.
I'm not sure why one would want to lock out the information that
This is a little off topic, as it doesn't relate directly to pinhole, but
many members of this list have websites exhibiting their work, so I decided
to put it out there. I have recently noticed a few requests every day on my
webserver for robots.txt. After getting that file (which doesn't even
