[
https://issues.apache.org/jira/browse/NUTCH-1300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225174#comment-13225174
]
Sebastian Nagel commented on NUTCH-1300:
----------------------------------------
+1
* effective fix for a serious problem: long running continuous crawls require
adjustments of the normalization rules quite often
* tested (with 1.4): costs (time spent for extra normalization) are ok compared
to the benefit
Two suggestions:
# Does a URLNormalizer scope "index" make sense? E.g., if only outlinks are
normalized and default rules are empty, the scope "index" may use the same
rules as scope "outlink".
# Wouldn't commandline options for solrindex be nice? Most other tools
(generate, updatedb, invertlinks) have options such as -filter / -norm /
-noNorm.
> Indexer to normalize URL's
> --------------------------
>
> Key: NUTCH-1300
> URL: https://issues.apache.org/jira/browse/NUTCH-1300
> Project: Nutch
> Issue Type: New Feature
> Components: indexer
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.5
>
> Attachments: NUTCH-1300-1.5-1.patch
>
>
> Indexers should be able to normalize URL's. This is useful when a new
> normalizer is applied to the entire CrawlDB. Without it, some or all records
> in a segment cannot be indexed at all.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
