Dear Wiki user, You have subscribed to a wiki page or wiki category on "Nutch Wiki" for change notification.
The "SimilarityScoringFilter" page has been changed by SujenShah: https://wiki.apache.org/nutch/SimilarityScoringFilter?action=diff&rev1=2&rev2=3 = Similarity based Scoring = - <<TableOfContents(5)>> + <<TableOfContents(6)>> == Summary == The link relevancy algorithm is based on the concept of the Cosine Similarity metric [1] to determine the semblance of fetched pages to accepted content in order to direct a NUTCH 1.X crawl in real-time. The score of how similar the text on a currently fetched page and that in user-specified gold-standard document are likely to be in terms of their subject matter is determined, the parsed page scored, and the all outlinks associated with that page scored accordingly. @@ -53, +53 @@ </property> }}} + + + == Example == + + The following is an example if how should thee goldstandard and stopwords file should look like. + + '''Gold Standard file''' + + This is a plain text file with no special formatting required. The contents of this file will be lowercased and split by a whitespace (Note: The tokenization process currently replaces any token which starts with a special character to a white space). A term frequency vector of the generated tokens is then created for computing the similarity. + + For example, if you wanted to score pages similar to the field of robotics, your goldstandard could look like: + + 1. Only keyterms + {{{ + Aerial Robotics Autonomous Agents Behaviour-Based Systems Brain Machine Interface Collision Avoidance Haptics and Haptic Interfaces ...... + }}} + + 2. Complete text (example text from a wikipedia article): + {{{ + An autonomous robot is a robot that performs behaviors or tasks with a high degree of autonomy, which is particularly desirable in fields such as space exploration, household maintenance (such as cleaning), waste water treatment and delivering goods and services. + + Some modern factory robots are "autonomous" within the strict confines of their direct environment. It may not be that every degree of freedom exists in their surrounding environment, but the factory robot's workplace is challenging and can often contain chaotic, unpredicted variables. The exact orientation and position of the next object of work and (in the more advanced factories) even the type of object and the required task must be determined. This can vary unpredictably (at least from the robot's point of view). + + One important area of robotics research is to enable the robot to cope with its environment whether this be on land, underwater, in the air, underground, or in space. + + A fully autonomous robot can + + Gain information about the environment + Work for an extended period without human intervention + Move either all or part of itself throughout its operating environment without human assistance + }}} + + '''Stopword File''' + + The stopword file should contain the words which you would like to omit while creating the term frequency vector. For example, the common terms in english grammar provide very little information about the relevance of a page and we would want them to be omitted. + + Each stopword should be on a new line (Note: Stopwords do not undergo the process of tokenization, that means they will be used as they appear in the stopwords.txt file). Even if only keywords are used to create the goldstandard file, a list of stopwords should be provided, as the fetched pages may contain those terms. + + Sample: + {{{ + a + about + above + after + again + against + all + am + an + and + any + are + aren't + as + at + be + because + been + before + being + below + between + both + but + by + can't + cannot + could + couldn't + did + didn't + do + .... + }}} + + == Future work == * Allow for other media formats in the gold-standard document
