Dear Andrzej,
This is very interesting patch, but I have a question:
- If page isn't modified, you don't refetch page.
- If you don't refrech page, there is in old segments?
- If it's in old segments, the segments data will be increasses, how to
analize that which segment is deletable?
- If it's in the old segments, the useable index will be larger and
larger. Because there is a limitation: optimal 2Kbyte RAM / page -> this
will decrease performance or increasse RAM usage?
Sorry my performance question, this patch is very interesting and usable.
Thanks for your your answer,
Ferenc
Andrzej Bialecki (JIRA) wrotte:
Adaptive re-fetch interval. Detecting umodified content
-------------------------------------------------------
Key: NUTCH-61
URL: http://issues.apache.org/jira/browse/NUTCH-61
Project: Nutch
Type: New Feature
Components: fetcher
Reporter: Andrzej Bialecki
Assigned to: Andrzej Bialecki
Currently Nutch doesn't adjust automatically its re-fetch period, no matter if
individual pages change seldom or frequently. The goal of these changes is to
extend the current codebase to support various possible adjustments to re-fetch
times and intervals, and specifically a re-fetch schedule which tries to adapt
the period between consecutive fetches to the period of content changes.
Also, these patches implement checking if the content has changed since last
fetching; protocol plugins are also changed to make use of this information, so
that if content is unmodified it doesn't have to be fetched and processed.
-------------------------------------------------------
This SF.Net email is sponsored by: NEC IT Guy Games. How far can you shotput
a projector? How fast can you ride your desk chair down the office luge track?
If you want to score the big prize, get to know the little guy.
Play to win an NEC 61" plasma display: http://www.necitguy.com/?r=20
_______________________________________________
Nutch-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
