[ http://issues.apache.org/jira/browse/NUTCH-61?page=all ]
Andrzej Bialecki updated NUTCH-61:
-----------------------------------
Attachment: 20050606.diff
The first round:
* change Page to use a 1-byte float, representing fetchInterval in seconds.
* implement a pluggable FetchSchedule, which adjusts fetchInterval and
nextFetchTime
* change FetchListTool and UpdateDatabaseTool to use them. NOTE: it appears
there was a bug in FetchListTool, where the fetchlist entries recorded in
segments would have their fetchTime increased by 1 week. This is not needed,
only pages in WebDB need this.
* improve status reporting throughout all plugins.
* change plugins to detect if the content is unchanged. If possible, plugins
will not fetch such content, but in any case they will set their status
accordingly.
> Adaptive re-fetch interval. Detecting umodified content
> -------------------------------------------------------
>
> Key: NUTCH-61
> URL: http://issues.apache.org/jira/browse/NUTCH-61
> Project: Nutch
> Type: New Feature
> Components: fetcher
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Attachments: 20050606.diff
>
> Currently Nutch doesn't adjust automatically its re-fetch period, no matter
> if individual pages change seldom or frequently. The goal of these changes is
> to extend the current codebase to support various possible adjustments to
> re-fetch times and intervals, and specifically a re-fetch schedule which
> tries to adapt the period between consecutive fetches to the period of
> content changes.
> Also, these patches implement checking if the content has changed since last
> fetching; protocol plugins are also changed to make use of this information,
> so that if content is unmodified it doesn't have to be fetched and processed.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
-------------------------------------------------------
This SF.Net email is sponsored by: NEC IT Guy Games. How far can you shotput
a projector? How fast can you ride your desk chair down the office luge track?
If you want to score the big prize, get to know the little guy.
Play to win an NEC 61" plasma display: http://www.necitguy.com/?r=20
_______________________________________________
Nutch-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
