[
https://issues.apache.org/jira/browse/NUTCH-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14595193#comment-14595193
]
Sebastian Nagel commented on NUTCH-2038:
----------------------------------------
Hi Chris, I would second Lewis that it's better to implement the functionality
as a ScoringFilter. The method
[distributeScoreToOutlink|https://nutch.apache.org/apidocs/apidocs-1.10/org/apache/nutch/scoring/ScoringFilter.html#distributeScoreToOutlinks%28org.apache.hadoop.io.Text,%20org.apache.nutch.parse.ParseData,%20java.util.Collection,%20org.apache.nutch.crawl.CrawlDatum,%20int%29]
is explicitly allowed to selectively drop outlinks. The URLFilter interface
does not allow for content-based dropping of URLs. That's why [~sujenshah]'s
recent patch requires changes to the interface and the ParseSegment class. An
implementation as ScoringFilter would allow to encapsulate the whole
functionality as plugin.
> Naive Bayes classifier based url filter
> ---------------------------------------
>
> Key: NUTCH-2038
> URL: https://issues.apache.org/jira/browse/NUTCH-2038
> Project: Nutch
> Issue Type: New Feature
> Components: fetcher, injector, parser
> Reporter: Asitang Mishra
> Assignee: Chris A. Mattmann
> Labels: memex, nutch
> Fix For: 1.11
>
>
> A url filter that will filter out the urls (after the parsing stage, will
> keep only those urls that contain some "hot words" provided again in a list.)
> from that pages that are classified irrelevant by the classifier.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
