[
https://issues.apache.org/jira/browse/NUTCH-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14609538#comment-14609538
]
Chris A. Mattmann commented on NUTCH-2038:
------------------------------------------
OK I found a few more issues:
1. Adding conf/naivebayes-*.txt.template didn't fully fix it, b/c the
conf/nutch-default.xml needed updating to the wordlist property [done]
2. TestFeedParser in parse-tika failed b/c the default training and wordlist
didn't identify the expected 2 outlinks as relevant. I've updated the
conf/naivebayes-wordlist.txt to address this.
3. the files generated by Mahout should be put e.g., into
crawl_dir/parsefilter-naivebayes, e.g.,
- temp
- vectors
- labelindex
- model
- outseq
- .labelindex.crc
[ to be done in a future patch, [~asitang] please file an issue for this]
4. Moved dependencies out of the plugin and into main Nutch ivy.xml [probably
not the best, but doesn't hurt too much for now and allows to get around the
plugin issue - done]
5. updated default.properties to declare the parsefilter-naivebayes [done]
Going to commit the update now. All tests pass locally and tested.
Thanks for everyone's review.
> Naive Bayes classifier based html Parse filter (for filtering outlinks)
> -----------------------------------------------------------------------
>
> Key: NUTCH-2038
> URL: https://issues.apache.org/jira/browse/NUTCH-2038
> Project: Nutch
> Issue Type: New Feature
> Components: fetcher, injector, parser
> Reporter: Asitang Mishra
> Assignee: Chris A. Mattmann
> Labels: memex, nutch
> Fix For: 1.11
>
>
> A html parse filter that will filter out the outlinks in two stages.
> Classify the parse text and decide if the parent page is relevant. If
> relevant then don't filter the outlinks. If irrelevant then go thru each
> outlink and see if the url contains any of the important words from a list.
> If it does then let it pass.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
