Jay,
The url field is handled by the query-basic filter. There is a
setting inside conf/nutch-default.xml that controls the weighting
(boost) for that field. You can reduce the influence of this field by
putting a new value in your conf/nutch-site.xml file. You may even be
able to completely nullify it by setting the value to 0.0. I've pasted
what I think you'd need to put in nutch-site.xml bellow. I haven't
tested this. Let me know how it goes if you give it a try.
Thanks,
Jake.
<property>
<name>query.url.boost</name>
<value>0.0</value>
<description> Used as a boost for url field in Lucene query.
</description>
</property>
-----Original Message-----
From: jay jiang [mailto:[EMAIL PROTECTED]
Sent: Thursday, February 16, 2006 2:08 PM
To: [email protected]
Subject: not indexing path names
I am crawling an intranet. Apparently Nutch also indexes the url path
names (as a document) as it crawls. So if a query word appears in the
path name, the entire url path name would be one result. Since this
kind of info would typically be of no value to users, I want to filter
them out.
I think we have to crawl them since we need to get the actual document
urls underneath the path. But we do not want to index them. Is there
anyway to configure not to index path names during the crawling step?
If not, can we configure it in the search step? I know we can always
filter it using getDetails(). But this seems not a very clean way.
Thanks,
--Jay
-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems? Stop! Download the new AJAX search engine that makes
searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid3432&bid#0486&dat1642
_______________________________________________
Nutch-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-general
