Markus as far as I see there is no CrawlDatum in nutch 2.1. However, it is interesting approach. What factors are considered by your classifier to detect hub pages? It parse urls or count outlinks?
2014-02-18 10:03 GMT+01:00 Mateusz Zakarczemny < [email protected]>: > As Jorge said it could be parametrized in seed file: > <URL>\tnutch.fetchInterval=86400 > It is quite important that if we use AdaptiveFetchSchedule interval will > be overriden. In nutch 1.6 it could be bypassed using > nutch.fetchInterval.fixed (Issue NUTCH-1388) but it wasn't yet ported to > nutch 2.1 (Issue NUTCH-1682) > > > > 2014-02-18 9:53 GMT+01:00 Markus Jelsma <[email protected]>: > > Hi >> >> We do something similar using a parse filter plugin and a custom >> scheduler. The parse filter plugin contains a SVM classifier that gives a >> high score to hub pages, or pages we consider not important, no content, >> overviews, lists etc. This score is passed back to the CrawlDatum and used >> in the scheduler to adjust fetch time partially based on the hub score. >> >> Markus >> >> -----Original message----- >> > From:Jorge Luis Betancourt González <[email protected]> >> > Sent: Tuesday 18th February 2014 0:48 >> > To: [email protected] >> > Subject: Re: Setting different fetch interval for some pages >> > >> > If I'm don't remember wrong in the list there was a patch to accomplish >> this, specifying the fetch interval in the seed file. Also this could work >> as a base to implement a custom plugin to accomplish your specific use case. >> > >> > ----- Original Message ----- >> > From: "Mateusz Zakarczemny" <[email protected]> >> > To: [email protected] >> > Sent: Monday, February 17, 2014 10:14:14 AM >> > Subject: Setting different fetch interval for some pages >> > >> > Hi, >> > >> > I'm going to crawl some set of news sites. Pages on those sites could be >> > divided into two types: category page and article page. I would like to >> > fetch categories pages more frequently than article pages. List of >> > categories is rather fixed so I could mark them manually. >> > >> > I know I could reach similar behaviour using AdaptiveFetchSchedule but >> it >> > require some time to adjust fetch time. This doesn't satisfy me because >> > before the fetch I already know how often pages should be re crawled. >> > >> > I wonder if it is possible in nutch to set different fetch intervals for >> > sites. I know that I could extend AbstractFetchSchedule and implement >> this >> > behaviour manually. This would require adding some extra field to >> WebPage >> > object which indicate what type of page we are dealing with. It is >> possible >> > to add such field to WebPage object? Maybe there is another approach? >> > >> > Regards, >> > Mateusz >> > >> ________________________________________________________________________________________________ >> > III Escuela Internacional de Invierno en la UCI del 17 al 28 de febrero >> del 2014. Ver www.uci.cu >> > >> > >
