Author: Alexander Barkov Email: [EMAIL PROTECTED] Message: > I really have no idea...It doesn't index all robot.txt files it finds, just some >.... cos it did index just 10 robots.txt files, which is very low compared to the 20 >to 30k domains that are in the index. > > Jan Paul
Probably those robots are not in the root directory of servers. indexer WILL index robots.txt under any non-root directory, e.g. http://servername/PATH/robots.txt Reply: <http://www.mnogosearch.org/board/message.php?id=3178> ___________________________________________ If you want to unsubscribe send "unsubscribe general" to [EMAIL PROTECTED]
