Dear Wiki user, You have subscribed to a wiki page or wiki category on "Nutch Wiki" for change notification.
The "SimilarityScoringFilter" page has been changed by SujenShah: https://wiki.apache.org/nutch/SimilarityScoringFilter New page: = Similarity based Scoring = <<TableOfContents(5)>> == Summary == The link relevancy algorithm is based on the concept of the Cosine Similarity metric [1] to determine the semblance of fetched pages to accepted content in order to direct a NUTCH 1.X crawl in real-time. The score of how similar the text on a currently fetched page and that in user-specified gold-standard document are likely to be in terms of their subject matter is determined, the parsed page scored, and the all outlinks associated with that page scored accordingly. == Implementation == The implementation leverages the Cosine Similarity Metric [1] to compute how relevant the currently fetched page is to a given crawl. The gold-standard document and current page are converted into the term frequency vectors D0 and D1 respectively. Stopword filtering is performed during the conversion to the term frequency vectors and the stopwords are user specified through the configuration file. The formula for calculating the Cosine similarity is then given as CosSim(D0,D1) = dot product of (vect(D0).vect(D1)) | vect(D0) | | vect(D1) | where |vect(X)| represents the Euclidean distance from the origin. The formula provides the cosine of the angle of separation between the two vectors and is bounded between cos(90) = 0 and cos(0) = 1 (inclusive). Hence, the smaller the angle, the more similar the vectors are, and the score tends to 1. The text-based features checked for during this score implementation of the Cosine Similarity Metric are: Parsed Text from the page The HTML meta-keywords The HTML meta-description == How to use == 1. Copy the gold-standard file into the conf directory and enter the name of this file in nutch-site.xml. {{{ <property> <name>scoring.similarity.model.path</name> <value>pathToFile</value> </property> }}} 2. Copy the stopword.txt file to the conf directory. The file should contain each word on a single line. Now, update the path in the nutch-site.xml file {{{ <property> <name>scoring.similarity.stopword.file</name> <value>pathToFile</value> </property> }}} 3. Enable the plugin by entering the following in the nutch-site.xml {{{ <property> <name>plugin.includes</name> <value>protocol-http|urlfilter-regex|parse-(text|html|js)|index-basic|query-(basic|site|url)|summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)</value> </property> }}} == Future work == * Allow for other media formats in the gold-standard document * Include text from title tags of images in the features checked during the Cosine Similarity Metric score calculation. * Allow user to input multiple documents as the gold-standard document * In Data cleaning task, perform tokenization of the documents using the Lucene tokenizer. In this case, use Lucene’s STOP_SET as the default stopword. * Currently all the outlinks are given from a given page are given the parent’s score. It is proposed to parse the anchor tag and the URL string of each outlink, and perform a weighted scoring for better relevancy and boosting the score of individual URL. * Allow user to enter a threshold for URL scores to be ingested added to the generator at the next round.
