I think that now this logic fails if the next fetch time for the page is already long in the past (i.e. older than FETCH_GENERATION_DELAY_MS). This may happen for various reasons, e.g. when you didn't generate fetchlists for more than 1 week, or you changed the minimum cutoff score. In such case the next fetch time may be set to a value already in the past, which will defeat this mechanism.
I think that line 507 should read:
507 page.setNextFetchTime(System.currentTimeMillis() + FETCH_GENERATION_DELAY_MS);
This means that the next fetch time will arrive exactly FETCH_GENERATION_DELAY_MS from now, unless this page will be updated from the fetched segment, at which moment this time will be reset based on the calculations of fetch time + fetch interval.
Anybody care to comment on this?
I think your analysis is correct. The existing code is buggy, and your suggested patch is a good one.
Long-term perhaps we will need to revisit this. Perhaps we should add another date, lastGeneratedForFetch. When this is more than, e.g., 7 days ago, we can regenerate the page. I hate to make Page larger, since its size directly impacts db performance. So perhaps we could also make both dates (nextFetch and lastGenerated) to use seconds instead of milliseconds, so that they will fit in an int.
Doug
------------------------------------------------------- This SF.Net email is sponsored by: Sybase ASE Linux Express Edition - download now for FREE LinuxWorld Reader's Choice Award Winner for best database on Linux. http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
