At 04:33 PM 11/18/2001 +0100, you wrote:
Hello,
I am building my database for the spider to fill but I have a problem.
SNIP
At first I thought about indexing only the words that seem relevants but
this way I can only make simple searches (ie : rabbit). Then I thought
about Indexing with the
http://www.hastingsresearch.com/net/06-anti-thesaurus.shtml
This is a proposal for a meta-tag to tell search engines to ignore certain
words on a page when scoring relevancy. Among other things, it mentions
robots.txt as problematic:
Also, returning to the robots.txt standard: it may be
