On 2010-03-11 15:53, nikinch wrote:
Hi everyone I've been using nutch for a while now and i've come up on a snag. I'm trying to find where new linked pages are added to the segment as a specific entry. To make myself clear i've been through the fetch class and the crawlDBFilter and reducer. But i'm looking for the initial entry where, for a given page, the links are transformed into segment entries, my objective here is to pass down te initial inject url to all it's liked pages. So when i create an entry for the linked urls of a wegbpage i'll add metadata to their definition giving them this originating url. By the time i get to CrawlDBFilter i already have entries for linked pages and lost the notion of which seed url brought us here. I thought the job would be done in the Fetcher maybe in the output function but i'm not finding where it happens. So if anyone knows and could point me in the right direction i'd appreciate it.
Currently the best place to do this is in your implementation of a ScoringFilter, in distributeScoreToOutlinks(). You can also modify one of the existing scoring plugins. I would advise against modifying the code directly in ParseOutputFormat, it's complex and fragile.
-- Best regards, Andrzej Bialecki <>< ___. ___ ___ ___ _ _ __________________________________ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: info at sigram dot com
