[
https://issues.apache.org/jira/browse/NUTCH-1614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13711435#comment-13711435
]
Markus Jelsma commented on NUTCH-1614:
--------------------------------------
Most if not all filters/normalizers support setting a config file so you can
use a different filter/normalize config file per stage. This way you can use a
different set of regex rules during fetch/update and another during indexing.
You'll have to check each plugin's code to know the exact configuration
parameter you need to point it to a different config file.
I think this was never ported to 2.x so i think it's better to first port the
pluggable indexing backends from 1.x to 2.x and then let it also support
filtering and normalizing.
Also, NUTCH-1300's title is wrong, it should be normalizing AND filtering. If
you check the patch you'll see it's actually about both.
> Plugin to exclude URLs matching regex list from indexing - to enable crawl
> but do not index
> -------------------------------------------------------------------------------------------
>
> Key: NUTCH-1614
> URL: https://issues.apache.org/jira/browse/NUTCH-1614
> Project: Nutch
> Issue Type: Improvement
> Components: indexer
> Affects Versions: 2.2.1
> Reporter: Brian
> Priority: Minor
> Labels: plugin
> Attachments: NUTCH-1614.patch
>
>
> Some pages we need to crawl (such as some main pages and different views of a
> main page) to get all the other pages, but we don't want to index those pages
> themselves. Therefore we cannot use the url filter approach.
> This plugin uses a file containing regex strings (see included sample file).
> If one of the regex strings matches with an entire URL, that URL will be
> excluded form indexing.
> The file to use is specified by the following property in nutch-site.xml:
> <property>
> <name>indexer.url.filter.exclude.regex.file</name>
> <value>regex-indexer-exclude-urls.txt</value>
> <description>
> Holds the file name containing the regex strings. Any URL
> matching one of these strings will be excluded from indexing.
> "#" indicates a comment line and will be ignored.
> </description>
> </property>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
