Disallows in robots.txt
So: I was looking at a robots.txt file and it had a series of disallow instructions for various user agents, and then at the bottom was a full disallow: - User-agent: aa Disallow: /cgi-bin Disallow: /stuff Disallow: /x.html User-agent: bb
Re: Disallows in robots.txt
It even overrides the other disalows. So in this robots.txt nothing is indexed! Bert >> Original Message << On 23/03/2000, 22:19:57, Jonathan Knoll <[EMAIL PROTECTED]> wrote regarding Disallows in robots.txt: > So: > I was looking at a robots.txt file and it ha