Author: Alexander Barkov Email: [EMAIL PROTECTED] Message: > while indexer creates robots.txt
What do you mean it creates robots.txt? indexer doesn't create any robots.txt. It fetches robots.txt from every site. Take a look here for an information about robots exclusion standards: http://www.robotstxt.org/wc/robots.html > these are not be placed in the table robots of my mySQL. Do you have any idea why >is this happening? Please give me your lights about it. Probably site you are indexing doesn't have robots.txt. Or robots.txt doesn't have any rules. Reply: <http://www.mnogosearch.org/board/message.php?id=3251> ___________________________________________ If you want to unsubscribe send "unsubscribe general" to [EMAIL PROTECTED]
