Hello!
Please send me your .conf file.
Regards, Ramil
----- Original Message -----
From: "Ole Dalgaard" <[EMAIL PROTECTED]>
To: <[EMAIL PROTECTED]>
Sent: Thursday, August 02, 2001 7:05 AM
Subject: Bug when indexing robots.txt with contenttype text/html and
status200
> When indexing anything from sites like
>
> http://www.subnet.dk/
> http://www.leisureplanet.com/
> http://www.ulitka.ru/
> http://go.to/
> http://welcome.to/
> http://come.to/
> http://www.cookinglight.com/
> http://www.ineedhits.com/
>
> you get a lot of other sites too. This is caused by their robots.txt
> files does not exist and/or return a text/html with http status 200,
> which causes mnogosearch 3.1.17 indexer to index all links on these
> pages with no restrictions. ( I use Follow path)
>
> Is it my configuration or is this a bug?
>
> BR
> Ole Dalgaard
>
> ___________________________________________
> If you want to unsubscribe send "unsubscribe general"
> to [EMAIL PROTECTED]
>
>
___________________________________________
If you want to unsubscribe send "unsubscribe general"
to [EMAIL PROTECTED]
