[
https://issues.apache.org/jira/browse/NUTCH-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14593982#comment-14593982
]
ASF GitHub Bot commented on NUTCH-2038:
---------------------------------------
Github user chrismattmann commented on a diff in the pull request:
https://github.com/apache/nutch/pull/34#discussion_r32870336
--- Diff: src/java/org/apache/nutch/parse/ParseSegment.java ---
@@ -69,6 +77,35 @@ public void configure(JobConf job) {
setConf(job);
this.scfilters = new ScoringFilters(job);
skipTruncated = job.getBoolean(SKIP_TRUNCATED, true);
+
+ filterflag = job.getBoolean(PARSER_MODELFILTER, true);
+ if (filterflag) {
+ String[] args = new String[2];
+ args[0] = getConf().get(TRAINFILE_MODELFILTER);
+ args[1] = getConf().get(DICTFILE_MODELFILTER);
+
+ if (args[0] == null || args[0].trim().length() == 0 || args[1] ==
null
+ || args[1].trim().length() == 0) {
+ String message = "Model URLFilter: trainfile or wordlist not set
in the urlfilter.model.trainfile or urlfilter.model.wordlist";
+ if (LOG.isErrorEnabled()) {
+ filterflag = false;
+ LOG.error(message);
+ }
+ throw new IllegalArgumentException(message);
+ } else {
+ try {
+ filters = new URLFilters(job);
+ filter = (ModelURLFilterAbstract) filters
--- End diff --
This ties us into using a specific filter, the ModelURLFilter, in the core
Nutch classes. Why can't the URL filter simply be insulated to the plugin -
this shouldn't have to touch the Nutch core?
> Naive Bayes classifier based url filter
> ---------------------------------------
>
> Key: NUTCH-2038
> URL: https://issues.apache.org/jira/browse/NUTCH-2038
> Project: Nutch
> Issue Type: New Feature
> Components: fetcher, injector, parser
> Reporter: Asitang Mishra
> Assignee: Chris A. Mattmann
> Labels: memex, nutch
> Fix For: 1.11
>
>
> A url filter that will filter out the urls (after the parsing stage, will
> keep only those urls that contain some "hot words" provided again in a list.)
> from that pages that are classified irrelevant by the classifier.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
