matt, how far along are you on this tool? i'm basically done (just need to do some more testing). sorry about the duplicated effort -- we needed this quickly so i just banged it out.
--- Matt Kangas <[EMAIL PROTECTED]> wrote: > Doug, thanks for the tips! I'll try to take a stab > at it ASAP. > > (Would try tonight, but I need to think about how > I'll validate the > output first...) > > --Matt > > On Thu, 06 Jan 2005 13:46:52 -0800, Doug Cutting > <[EMAIL PROTECTED]> wrote: > > Matt Kangas wrote: > > > Hi Doug, I've taken a quick look at this tonight > to see if I > > > understand all of the relevant concepts. I think > I'm about halfway > > > there. ;-) But I do have a few questions: > > > > Great! > > > > >> - iterate through fetcher/ data, generating a > <url,segment,n> output > > >>file, where n is the url's position within the > segment. > > >> - sort this file by url. > > > > > > > > > Would there be value in doing this on a > per-segment basis instead of > > > multi-segment? E.g. creating an output file > <url,n> within the segment > > > directory. > > > > Processing everything at once is optimal in terms > of CPU and io, but > > requires more temporary disk space. So, if disk > space is tight then > > segment-at-a-time might be preferable. > > > > >> - merge the sorted file with the db, > generating a > > >><segment,n,inlinks,scores> file. > > >> - sort this file by <segment,n>. > > > > > > > > > <inlinks,scores> comes from the webdb, right? > > > > Right. Inlinks are really just anchors. (Finding > the incoming url is > > considerably more expensive, and isn't required.) > > > > >> - merge this with the segment's fetcher/ > files, rewriting each with > > >>updated inlinks and score. > > > > > > I'm a bit fuzzy on this one. Where are inlinks > and score stored in a > > > segment? FetchListEntry.anchors and > FetchList.page.score/nextScore? > > > > Yup. nextScore is only used internally in > link-analysis, so all that > > you need to update is the anchors and score. > > > > Doug > > > > > ------------------------------------------------------- > The SF.Net email is sponsored by: Beat the > post-holiday blues > Get a FREE limited edition SourceForge.net t-shirt > from ThinkGeek. > It's fun and FREE -- well, > almost....http://www.thinkgeek.com/sfshirt > _______________________________________________ > Nutch-developers mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/nutch-developers > __________________________________ Do you Yahoo!? The all-new My Yahoo! - Get yours free! http://my.yahoo.com ------------------------------------------------------- The SF.Net email is sponsored by: Beat the post-holiday blues Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek. It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt _______________________________________________ Nutch-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-developers
