So:
I was looking at a robots.txt file and it had a series of disallow
instructions for various user agents, and then at the bottom was a full
disallow:
[...]
Wouldn't this just disallow everyone from everything?
No, it would disallow everyone but a ... d (with the
specified
Jonathan Knoll:
User-agent: *
Disallow: /cgi-bin
Disallow: /site
Klaus Johannes Rusch:
/cgi-bin/test.cgi
/siteindex.html
would be excluded.
But what about these paths (in the same root dir):
/foo/cgi-bin/test.cgi
/bar/user1/cgi-bin/test.sgi
