Author: Orjan Sandland
Email: [EMAIL PROTECTED]
Message:
I had alot of problems with my indexer trying to index mailto cgi's, search cgi's and
forward/backwards links in documents.
This is related to discussion forums like UBB and others.
I got some advice on how to filter certain cgi's with options, but I think I must have
done something seriously wrong!
Before I modified the indexer.conf, I had 300.000 URLs in the database. Now I've got
2500!!!!
If someone could check out the following section of my indexer.conf - I'd be most
thankful:
Disallow *out.cgi
Disallow *privatesend.cgi *action*
Disallow *ubbmisc.cgi *findthread*
Disallow *search.cgi *simplesearch*
Disallow *Ultimate.cgi *email*
Disallow *ultimatebb.cgi *get_ip*
Disallow *ultimatebb.cgi *reply*
Disallow *ultimatebb.cgi *send_topic*
Disallow *ultimatebb.cgi *next_topic*
Disallow *ultimatebb.cgir *edit_post*
Disallow *ultimatebb.cgi *close_topic*
Disallow *ultimatebb.cgi *email*
Disallow *ultimatebb.cgi *delete_topic*
#
Disallow NoMatch String *www.rc-racing.com/*backtalk/pistachio*
Disallow NoMatch String *www.rc-racing.com/*backtalk/abalone*
Alot of URL processing are now down the drain. I need to fix the problem and index the
whole thing over again :-(.
I can add that I'm not using robots.txt.... (it seemed to stop alot of relevant URLs
getting inexed) but other than that - my config is a copy of the sample.
Running multi-crc mode.
Thanks in advance.
Best regards,
�rjan Sandland
Senior Consultant
Net Technology AS
Reply: <http://search.mnogo.ru/board/message.php?id=2101>
___________________________________________
If you want to unsubscribe send "unsubscribe general"
to [EMAIL PROTECTED]
