7. 16 INTERVAL (ushort) refetch frequency in hours (max 7.6 years)
I think we might reasonably need to support sub-hour refresh, e.g., for blogs and perhaps news. We could either make this a uint32, or use a float number of days with, e.g., eight bits of mantissa, eight bits of exponent and no sign bit. Lucene has an eight-bit float format (three bits mantissa, five bits exponent) which would probably be sufficient.
http://cvs.apache.org/viewcvs.cgi/*checkout*/jakarta-lucene/src/java/org/apache/lucene/search/Similarity.java?rev=1.17
However, there are two distinct parts in this structure - one is fixed, the other is variable over time. Fields 1-3 are fixed (for a given version of Nutch software). Fields 4-9 are the ones being updated over time. Let's call them part1 and part2:
- part1: fields 1-3, size: 29 bytes or more - part2: fields 4-9, size: 19 bytes
The information from part1 will never change (except when we change the VERSION), so it doesn't make much sense to copy it around when doing updates... and its size is enormous compared to the size of part2..
As one crawls one continually finds new pages and links, as well as new versions of existing urls. As long as the rate of new pages and links is significant, one will have to perform b-tree inserts which will fragment a B-tree. I think collections which only change content and never the set of urls are quite rare. So I am not optimistic about this avenue of optimization.
Is db update really slower than crawling for ~20M page collections? It should not be. If it is, then let's do some profiling and optimize it.
Doug
------------------------------------------------------- This SF.Net email is sponsored by: Sybase ASE Linux Express Edition - download now for FREE LinuxWorld Reader's Choice Award Winner for best database on Linux. http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
