Bert Van Kets:
It even overrides the other disalows.
No override... Robots Exclusion Standard on User-agent:
(If the value is '*', the record describes the default access policy for
any robot that) has not [yet] matched any of the other records.
So:
I was looking at a robots.txt file and it had a series of disallow
instructions for various user agents, and then at the bottom was a full
disallow:
[...]
Wouldn't this just disallow everyone from everything?
No, it would disallow everyone but a ... d (with the
specified
