[EMAIL PROTECTED] wrote:
Dear Andrzej,
This is very interesting patch, but I have a question:
- If page isn't modified, you don't refetch page.
Correct - that's the whole point of this patch.
- If you don't refrech page, there is in old segments?
Yes, the page should be in some old segment. This question brings an
interesting dilemma - should I add an option to forcefully refetch a
page, in case you lost the old segment data? Hmmm... In the current code
there is an option "-adddays", but with adjustable interval this doesn't
make much sense.
- If it's in old segments, the segments data will be increasses, how to
analize that which segment is deletable?
Well, there is no good answer to that even with the current code... You
can use mergesegs tool to keep only the latest versions of pages. But I
agree, this patch make the problem of handling old segments more serious
- how to "phase out" older segments.
- If it's in the old segments, the useable index will be larger and
larger. Because there is a limitation: optimal 2Kbyte RAM / page -> this
will decrease performance or increasse RAM usage?
The index (Lucene index) will not be larger - the deduplication process
takes care of that. Only the latest version of the content will show up
in the index, and for identical content only the one reachable via the
shortest URL.
Sorry my performance question, this patch is very interesting and usable.
Thanks for good, thought-provoking questions!
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
-------------------------------------------------------
This SF.Net email is sponsored by: NEC IT Guy Games. How far can you shotput
a projector? How fast can you ride your desk chair down the office luge track?
If you want to score the big prize, get to know the little guy.
Play to win an NEC 61" plasma display: http://www.necitguy.com/?r=20
_______________________________________________
Nutch-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
