[
https://issues.apache.org/jira/browse/NUTCH-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14593983#comment-14593983
]
ASF GitHub Bot commented on NUTCH-2038:
---------------------------------------
Github user chrismattmann commented on a diff in the pull request:
https://github.com/apache/nutch/pull/34#discussion_r32870349
--- Diff: src/java/org/apache/nutch/parse/ParseSegment.java ---
@@ -140,6 +177,37 @@ public void map(WritableComparable<?> key, Content
content,
LOG.warn("Error passing score: " + url + ": " + e.getMessage());
}
}
+
+ if (filterflag) {
+
+ if (!filter.filterParse(parse.getText())) { // kick in the second
tier
+ // if parent page found
+ // irrelevent
+ LOG.info("ModelURLFilter: Page found irrelevent:: " + url);
--- End diff --
needs to be insulated.
> Naive Bayes classifier based url filter
> ---------------------------------------
>
> Key: NUTCH-2038
> URL: https://issues.apache.org/jira/browse/NUTCH-2038
> Project: Nutch
> Issue Type: New Feature
> Components: fetcher, injector, parser
> Reporter: Asitang Mishra
> Assignee: Chris A. Mattmann
> Labels: memex, nutch
> Fix For: 1.11
>
>
> A url filter that will filter out the urls (after the parsing stage, will
> keep only those urls that contain some "hot words" provided again in a list.)
> from that pages that are classified irrelevant by the classifier.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
