Re: Issue in generating URLs for re-fetching once db.fetch.interval.max elapses

[vivekvl]

Hi Lewis,
I am expecting this method to return true when maxInterval elapses for a
page, so that it could be included in the generate list.

@Override
public boolean shouldFetch(String url, WebPage page, long curTime) {
  // pages are never truly GONE - we have to check them from time to time.
  // pages with too long fetchInterval are adjusted so that they fit within
  // maximum fetchInterval (segment retention period).
  long fetchTime = page.getFetchTime();
  if (fetchTime - curTime > maxInterval * 1000L) {
    if (page.getFetchInterval() > maxInterval) {
      page.setFetchInterval(Math.round(maxInterval * 0.9f));
    }
    page.setFetchTime(curTime);
  }
  return fetchTime <= curTime;
}



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Issue-in-generating-URLs-for-re-fetching-once-db-fetch-interval-max-elapses-tp4079039p4079343.html
Sent from the Nutch - User mailing list archive at Nabble.com.