When indexing anything from sites like
http://www.subnet.dk/
http://www.leisureplanet.com/
http://www.ulitka.ru/
http://go.to/
http://welcome.to/
http://come.to/
http://www.cookinglight.com/
http://www.ineedhits.com/
you get a lot of other sites too. This is caused by their robots.txt
files does not exist and/or return a text/html with http status 200,
which causes mnogosearch 3.1.17 indexer to index all links on these
pages with no restrictions. ( I use Follow path)
Is it my configuration or is this a bug?
BR
Ole Dalgaard
___________________________________________
If you want to unsubscribe send "unsubscribe general"
to [EMAIL PROTECTED]
