Hi Doug, I've taken a quick look at this tonight to see if I understand all of the relevant concepts. I think I'm about halfway there. ;-) But I do have a few questions:
Great!
- iterate through fetcher/ data, generating a <url,segment,n> output file, where n is the url's position within the segment. - sort this file by url.
Would there be value in doing this on a per-segment basis instead of multi-segment? E.g. creating an output file <url,n> within the segment directory.
Processing everything at once is optimal in terms of CPU and io, but requires more temporary disk space. So, if disk space is tight then segment-at-a-time might be preferable.
- merge the sorted file with the db, generating a <segment,n,inlinks,scores> file. - sort this file by <segment,n>.
<inlinks,scores> comes from the webdb, right?
Right. Inlinks are really just anchors. (Finding the incoming url is considerably more expensive, and isn't required.)
- merge this with the segment's fetcher/ files, rewriting each with updated inlinks and score.
I'm a bit fuzzy on this one. Where are inlinks and score stored in a segment? FetchListEntry.anchors and FetchList.page.score/nextScore?
Yup. nextScore is only used internally in link-analysis, so all that you need to update is the anchors and score.
Doug
------------------------------------------------------- The SF.Net email is sponsored by: Beat the post-holiday blues Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek. It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt _______________________________________________ Nutch-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-developers
