> So:
> I was looking at a robots.txt file and it had a series of disallow
> instructions for various user agents, and then at the bottom was a full
> disallow:
> Wouldn't this just disallow everyone from everything?

No, it would disallow everyone but aaaaa ... ddddd (with the
specified restrictions).

>From the spec:
"  The robot
   must obey the first record in /robots.txt that contains a User-
   Agent line whose value contains the name token of the robot as a
   substring. The name comparisons are case-insensitive. If no such
   record exists, it should obey the first record with a User-agent
   line with a "*" value, if present. If no record satisfied either
   condition, or no records are present at all, access is unlimited."

Regards, Martin

Reply via email to