"Colin J. Raven" wrote:
> On Thu, 8 Mar 2001, Siao Yuan Tan wrote:
> > Inside my Cobalt RaQ4r, i found this file robots.txt in /usr/admserv/html/
> > folder with the following content:
> >
> > # Prevent all robots from visiting this site:
> >
> > User-agent: *
> > Disallow: /
> >
> > I come to know this file from the webalizer report because this file seem to
> > have a number of hits to it. Anyone know what is this file doing in my
> > server?
>
> It's there to prevent spiders from roving through (and reporting on) your
> admin pages. .....<snip>
Can I suggest it this is not in fact the case ! - meant nicely :)
It is really there to ***ASK*** robots / spiders not to perform the search (or used in
a normal context, to request limits as to what gets searched i.e. "Disallow /images"
is a common example).
"Rude" robots will not even bother looking at the file <g>
I know its only symatics, but it is important for all to realise that this method will
"NOT" stop nosey robots !
--
Regards,
Kul
_______________________________________________
cobalt-security mailing list
[EMAIL PROTECTED]
http://list.cobalt.com/mailman/listinfo/cobalt-security
