[
https://issues.apache.org/jira/browse/NUTCH-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14599748#comment-14599748
]
Chris A. Mattmann commented on NUTCH-2038:
------------------------------------------
yeah you got it Seb, we can do accept/reject by threshold. See:
http://github.com/chrismattmann/tika-img-similarity/ where I have been already
doing this for a while and my search engines class
http://sunset.usc.edu/classes/cs572_2015/ specifically HW1 where I had them
develop something similar. The idea would be to use it to find similar objects
with features, and to accept those e.g., that would fall within a threshhold.
There is no difference with Jaccard, Cosine, Edit Distance, whatever. They are
all simply distance measurements. They can be used in Search Engines
deduplication; in scoring; in URL filtering, in a number of places. Anyways
I'll try and get something up soon.
In the meanwhile I am +1 for Asitang's latest PR, modulo my stylistic updates I
suggested.
Thanks for the great feedback as usual.
> Naive Bayes classifier based url filter
> ---------------------------------------
>
> Key: NUTCH-2038
> URL: https://issues.apache.org/jira/browse/NUTCH-2038
> Project: Nutch
> Issue Type: New Feature
> Components: fetcher, injector, parser
> Reporter: Asitang Mishra
> Assignee: Chris A. Mattmann
> Labels: memex, nutch
> Fix For: 1.11
>
>
> A url filter that will filter out the urls (after the parsing stage, will
> keep only those urls that contain some "hot words" provided again in a list.)
> from that pages that are classified irrelevant by the classifier.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
