I'm using FetchListTool, but in a slightly less usual way, so I haven't seen the issue you are describing, but yes, the current code does look wrong. One could get even a little defensive and add a check for page.nextFetchTime() and do 'now + interval' only if that nextTextTime returns something before(now).
Otis --- Andrzej Bialecki <[EMAIL PROTECTED]> wrote: > Hi, > > Reading the code in FetchListTool I found the following snippet which > > puzzles me: > > 494 // > 495 // Last, add the FetchListEntry to a file so we can > 496 // sort by score. Be sure to modify the Page's > 497 // fetchtime; this allows us to soon generate > 498 // another fetchlist which would not include this > 499 // Page. That's helpful because with two distinct > 500 // fetchlists, it should be possible to fetch and > 501 // perform dbupdate at the same time. > 502 // > 503 // Optionally set the score by the log of number of > 504 // incoming anchors. > 505 curScore.set(scoreByLinkCount ? > 506 (float)Math.log(results.length) : page.getScore()); > 507 page.setNextFetchTime(page.getNextFetchTime() + > FETCH_GENERATION_DELAY_MS); > > The whole logic of setting the next fetch time in the future is > dubious > - I think the fetchtime calculations will be more complex when/if we > implement the adaptive fetch interval, and this value should not be > abused - currently it acts as a flag to prevent the same page to be > fetched twice, if we run this tool twice without updating the DB in > the > meantime. But assuming we want it to work as it is... > > I think that now this logic fails if the next fetch time for the page > is > already long in the past (i.e. older than FETCH_GENERATION_DELAY_MS). > > This may happen for various reasons, e.g. when you didn't generate > fetchlists for more than 1 week, or you changed the minimum cutoff > score. In such case the next fetch time may be set to a value already > in > the past, which will defeat this mechanism. > > I think that line 507 should read: > > 507 page.setNextFetchTime(System.currentTimeMillis() + > FETCH_GENERATION_DELAY_MS); > > This means that the next fetch time will arrive exactly > FETCH_GENERATION_DELAY_MS from now, unless this page will be updated > from the fetched segment, at which moment this time will be reset > based > on the calculations of fetch time + fetch interval. > > Anybody care to comment on this? > > (BTW. I think it would be very nice to introduce a batch history to > WebDB - i.e. to keep a log of when new fetchlists have been > generated, > when the DB was updated, and so on. It would help a lot to track the > current status of fetching/updating. Also, we could then assign the > last > batch ID to each page, to be reset on successful update - this way we > > could avoid messing up with fetchtime, and we could just invalidate > the > old batches in order to "unlock" pages for fetching). > > -- > Best regards, > Andrzej Bialecki > > ------------------------------------------------- > Software Architect, System Integration Specialist > CEN/ISSS EC Workshop, ECIMF project chair > EU FP6 E-Commerce Expert/Evaluator > ------------------------------------------------- > FreeBSD developer (http://www.freebsd.org) > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by: > Sybase ASE Linux Express Edition - download now for FREE > LinuxWorld Reader's Choice Award Winner for best database on Linux. > http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click > _______________________________________________ > Nutch-developers mailing list > [EMAIL PROTECTED] > https://lists.sourceforge.net/lists/listinfo/nutch-developers > ------------------------------------------------------- This SF.Net email is sponsored by: Sybase ASE Linux Express Edition - download now for FREE LinuxWorld Reader's Choice Award Winner for best database on Linux. http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
