Author: Alexander Barkov
Email: [EMAIL PROTECTED]
Message:
As far as I know whildcards like * are not supported by robots.txt
standard. And they are not implemented in msearch.
> I had alot of problems with my indexer trying to index mailto cgi's, search cgi's
>and forward/backwards links in documents.
>
> This is related to discussion forums like UBB and others.
>
> I got some advice on how to filter certain cgi's with options, but I think I must
>have done something seriously wrong!
> Before I modified the indexer.conf, I had 300.000 URLs in the database. Now I've got
>2500!!!!
>
> If someone could check out the following section of my indexer.conf - I'd be most
>thankful:
>
> Disallow *out.cgi
> Disallow *privatesend.cgi *action*
> Disallow *ubbmisc.cgi *findthread*
> Disallow *search.cgi *simplesearch*
> Disallow *Ultimate.cgi *email*
> Disallow *ultimatebb.cgi *get_ip*
> Disallow *ultimatebb.cgi *reply*
> Disallow *ultimatebb.cgi *send_topic*
> Disallow *ultimatebb.cgi *next_topic*
> Disallow *ultimatebb.cgir *edit_post*
> Disallow *ultimatebb.cgi *close_topic*
> Disallow *ultimatebb.cgi *email*
> Disallow *ultimatebb.cgi *delete_topic*
> #
> Disallow NoMatch String *www.rc-racing.com/*backtalk/pistachio*
> Disallow NoMatch String *www.rc-racing.com/*backtalk/abalone*
>
>
> Alot of URL processing are now down the drain. I need to fix the problem and index
>the whole thing over again :-(.
>
> I can add that I'm not using robots.txt.... (it seemed to stop alot of relevant URLs
>getting inexed) but other than that - my config is a copy of the sample.
> Running multi-crc mode.
>
> Thanks in advance.
>
> Best regards,
> �rjan Sandland
> Senior Consultant
> Net Technology AS
>
Reply: <http://search.mnogo.ru/board/message.php?id=2103>
___________________________________________
If you want to unsubscribe send "unsubscribe general"
to [EMAIL PROTECTED]
