Doug Cutting wrote:
Long-term perhaps we will need to revisit this. Perhaps we should add another date, lastGeneratedForFetch. When this is more than, e.g., 7 days ago, we can regenerate the page. I hate to make Page larger, since its size directly impacts db performance. So perhaps we could also make both dates (nextFetch and lastGenerated) to use seconds instead of milliseconds, so that they will fit in an int.
I understand that Page size is crucial, as are any other file-based operations that need to change a lot of data.
Currently the Page looks like this:
8 VERSION (byte) var URL (String) 128 ID (hash) 64 DATE (long) 8 RETRIES (byte) 8 INTERVAL (byte) 32 SCORE (float) 32 NEXTSCORE (float)
Currently, this amounts to 35 bytes + variable amount for the URL (but minimum of 2 + "xxx://x.xx" = 12 bytes) = minimum 47 bytes.
Let's assume we change it to this:
1. 8 VERSION (byte) 2. var URL (String) URL 3. 128 ID (hash) content hash 4. 32 NEXTFETCH(uint) next fetch date, rounded to seconds 5. 32 LASTGEN (uint) last generated date, rounded to seconds 6. 8 RETRIES (byte) number of failed attempts 7. 16 INTERVAL (ushort) refetch frequency in hours (max 7.6 years) 8. 32 SCORE (float) current score 9. 32 NEXTSCORE (float) next score (BTW. what's the difference?)
This Page weighs at 36 bytes + variable URL = minimum 48 bytes. INTERVAL had to be increased, because the current resolution is too small to be used as a basis for adaptive interval adjustment.
However, there are two distinct parts in this structure - one is fixed, the other is variable over time. Fields 1-3 are fixed (for a given version of Nutch software). Fields 4-9 are the ones being updated over time. Let's call them part1 and part2:
- part1: fields 1-3, size: 29 bytes or more - part2: fields 4-9, size: 19 bytes
The information from part1 will never change (except when we change the VERSION), so it doesn't make much sense to copy it around when doing updates... and its size is enormous compared to the size of part2..
I'm thinking aloud now... would it improve the speed if we keep the part1 information in a separate structure, perhaps a b-tree (since it needs to be accessed quickly, and needs to be sorted, but rarely updated by inserting/removing nodes), and then we could do the frequent updates on the files containing only part2 info?
-- Best regards, Andrzej Bialecki
------------------------------------------------- Software Architect, System Integration Specialist CEN/ISSS EC Workshop, ECIMF project chair EU FP6 E-Commerce Expert/Evaluator ------------------------------------------------- FreeBSD developer (http://www.freebsd.org)
------------------------------------------------------- This SF.Net email is sponsored by: Sybase ASE Linux Express Edition - download now for FREE LinuxWorld Reader's Choice Award Winner for best database on Linux. http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
