"Christian Biesinger" <[EMAIL PROTECTED]> schrieb im Newsbeitrag [EMAIL PROTECTED]">news:[EMAIL PROTECTED]... > [EMAIL PROTECTED] wrote: > > Aaron Andersen wrote: > >> And one of the rules (although perhaps not written out because it is > >> so completely obvious) is that a user agent doesn't go and ask web > >> servers for random files just because there might possibly be a file > >> with that name that would be useful. It just isn't right. > > robots.txt?
> Are you saying that Mozilla downloads robots.txt? No, but another user agent does: the search engine.
