- - - - - - - - - - - - - - - - - - - - - - - - - - - - Name: mico Subject: Re: robots.txt:
Thank you for that... I just tested and it works. Unfortunatly... All it changed is the message that now says: "robots.txt support is disallowed for 'my.site.com'" The result is the same. Example: I try to get 'htp://digg.com/technology/' directory. Configuration: Robots no Server http://digg.com/technology/ Result: indexer[16276]: {01} URL: http://digg.com/view/technology/ indexer[16276]: {01} robots.txt support is disallowed for 'digg.com' indexer[16276]: {01} Done (2 seconds, 1 documents, 609 bytes, 0.30 Kbytes/sec.) indexer[16276]: {00} Total 3 seconds, 1 documents, 609 bytes, 0.20 Kbytes/sec, 3.00 sec/doc, 609 bytes/doc. Conclusion: This site does have links to subdirectories under technology/ so it should take those pages, right? If i try the configuration on another site that doesn't have robot.txt it works fine... Is this dataparksearch problem or am i mis-configuring something ? Thanks, mico - - - - - - - - - - - - - - - - - - - - - - - - - - - - Read the full topic here: http://www.dataparksearch.org/cgi-bin/simpleforum.cgi?fid=02;topic_id=1185818418
