Well I have created for my own application is topical-scoring plugin : 1. first I needed to score the pages after parsing based on my regular expression
2. then I searched several options on to how boost score of that pages... I have only found a way to boost the score of the outlinks of these pages that have content which I wanted. Not perfect but so be it there is a high likelyhood in my case that adjacent pages have also content which I want. 3. then how to boost the score... this took me a while to figure out, I leave you all the options I tried. The good comprise I found is the following: if the page has content I want and score < 1.0f than score= squareroot(score)... in this way you are adding weight to the pages which have content you are looking (since score is usually below 1. squareroot(x) is bigger than x). Of course there are some down side to that approach, it is more difficult to get the crawler to go outsides sites that have content your are looking for, it is a bit like digging a hole and until you have finished the hole it will get the crawler to explore it... experimentally I have found that it works nicely for me though, if you limit the nbre of URLS per site it won't spend it's life on them. We could try to generalize this plug-in by putting the regular expression as as config item because that is really the only thing which is specific to my application I believe. 2009/11/27 Eran Zinman <[email protected]> > Hi all, > > I'm try to figure out ways to improve Nutch focused crawling efficiency. > > I'm looking for certain pages inside each domain which contains content I'm > looking for. > > I'm unable to know that a certain URL contains what I'm looking for unless > I > parse it and do some analysis on it. > > Basically I was thinking about two methods to improve crawling efficiency: > > 1) Whenever a page is found which contains the data I'm looking for, > improve > overall score for all pages linking to it (and pages linking to them and so > on...), assuming they have other links that point to content I'm looking > for. > 2) Once I already found several pages that contain relevant data - create a > Regex automatically to match new urls which might contain usable content. > > I've started to read about the OPIC-score plugin but was unable to > understand if it can help me or not with issue no. 1. > > Any idea guys? I will be very grateful for any help or things that can > point > me in the right direction. > > Thanks, > Eran > -- -MilleBii-
