Disallows in robots.txt

2000-03-23 Thread Jonathan Knoll
So: I was looking at a robots.txt file and it had a series of disallow instructions for various user agents, and then at the bottom was a full disallow: - User-agent: aa Disallow: /cgi-bin Disallow: /stuff Disallow: /x.html User-agent: bb

Re: Disallows in robots.txt

2000-03-23 Thread Bert Van Kets
It even overrides the other disalows. So in this robots.txt nothing is indexed! Bert >> Original Message << On 23/03/2000, 22:19:57, Jonathan Knoll <[EMAIL PROTECTED]> wrote regarding Disallows in robots.txt: > So: > I was looking at a robots.txt file and it ha