Doug, thanks for the tips! I'll try to take a stab at it ASAP. (Would try tonight, but I need to think about how I'll validate the output first...)
--Matt On Thu, 06 Jan 2005 13:46:52 -0800, Doug Cutting <[EMAIL PROTECTED]> wrote: > Matt Kangas wrote: > > Hi Doug, I've taken a quick look at this tonight to see if I > > understand all of the relevant concepts. I think I'm about halfway > > there. ;-) But I do have a few questions: > > Great! > > >> - iterate through fetcher/ data, generating a <url,segment,n> output > >>file, where n is the url's position within the segment. > >> - sort this file by url. > > > > > > Would there be value in doing this on a per-segment basis instead of > > multi-segment? E.g. creating an output file <url,n> within the segment > > directory. > > Processing everything at once is optimal in terms of CPU and io, but > requires more temporary disk space. So, if disk space is tight then > segment-at-a-time might be preferable. > > >> - merge the sorted file with the db, generating a > >><segment,n,inlinks,scores> file. > >> - sort this file by <segment,n>. > > > > > > <inlinks,scores> comes from the webdb, right? > > Right. Inlinks are really just anchors. (Finding the incoming url is > considerably more expensive, and isn't required.) > > >> - merge this with the segment's fetcher/ files, rewriting each with > >>updated inlinks and score. > > > > I'm a bit fuzzy on this one. Where are inlinks and score stored in a > > segment? FetchListEntry.anchors and FetchList.page.score/nextScore? > > Yup. nextScore is only used internally in link-analysis, so all that > you need to update is the anchors and score. > > Doug > ------------------------------------------------------- The SF.Net email is sponsored by: Beat the post-holiday blues Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek. It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt _______________________________________________ Nutch-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-developers
