> So:
>
> I was looking at a robots.txt file and it had a series of disallow
> instructions for various user agents, and then at the bottom was a full
> disallow:
[...]
> Wouldn't this just disallow everyone from everything?
No, it would disallow everyone but a ... d (with the
specified
Bert Van Kets:
>It even overrides the other disalows.
No override... Robots Exclusion Standard on User-agent:
"(If the value is '*', the record describes the default access policy for
any robot that) has not [yet] matched any of the other records."
(http://info.webcrawler.com/mak/projects/robots/
Jonathan Knoll:
>User-agent: aa
>Disallow: /cgi-bin
>Disallow: /stuff
>Disallow: /x.html
[...]
>User-agent: *
>Disallow: /
>Wouldn't this just disallow everyone from everything?
No, the file is perfectly OK...
The "*" has a special meaning in the Standard:
"every other User-agent (
It even overrides the other disalows.
So in this robots.txt nothing is indexed!
Bert
>>>>>>>>>>>>>>>>>> Original Message <<<<<<<<<<<<<<<<<<
On 23/03/2000, 22:19:57, Jonathan Knoll <[E
So:
I was looking at a robots.txt file and it had a series of disallow
instructions for various user agents, and then at the bottom was a full
disallow:
-
User-agent: aa
Disallow: /cgi-bin
Disallow: /stuff
Disallow: /x.html
User-agent: bb
