- - - - - - - - - - - - - - - - - - - - - - - - - - - -
Name: mico
Subject: Re: robots.txt:
Results:
--------
Aug 2 11:51:07 prole indexer[17364]: {01} Allow method is used
Aug 2 11:51:07 prole indexer[17364]: {01} No conditional subsection detected
Aug 2 11:51:07 prole indexer[17364]: {01} Link '/' http://digg.com/
Aug 2 11:51:07 prole indexer[17364]: {01} Allow by default
Aug 2 11:51:07 prole indexer[17364]: {01} no Server, skip it
Aug 2 11:51:07 prole indexer[17364]: {01} Link '/about' http://digg.com/about
Aug 2 11:51:07 prole indexer[17364]: {01} Disallow Regex Sensitive
'\/$|\.htm$|\.html$|\.shtml$|\.phtml$|\.php$|\.txt$|\.asp$', skip it
Aug 2 11:51:07 prole indexer[17364]: {01} Link '/about-rss'
http://digg.com/about-rss
Aug 2 11:51:07 prole indexer[17364]: {01} Disallow Regex Sensitive
'\/$|\.htm$|\.html$|\.shtml$|\.phtml$|\.php$|\.txt$|\.asp$', skip it
Aug 2 11:51:07 prole indexer[17364]: {01} Link '/add-digg'
http://digg.com/add-digg
My Regex wasn't good. So I added to indexer.conf :
Allow *
:) And now it works fine, thanks! :)
But, if i put "Allow *" and right after that "Disallow *.cgi", will it still
exclude *.cgi or will it just directly take everything?
- - - - - - - - - - - - - - - - - - - - - - - - - - - -
Read the full topic here:
http://www.dataparksearch.org/cgi-bin/simpleforum.cgi?fid=02;topic_id=1185818418
