Hi Doug, I've taken a quick look at this tonight to see if I understand all of the relevant concepts. I think I'm about halfway there. ;-) But I do have a few questions:
> - iterate through fetcher/ data, generating a <url,segment,n> output > file, where n is the url's position within the segment. > - sort this file by url. Would there be value in doing this on a per-segment basis instead of multi-segment? E.g. creating an output file <url,n> within the segment directory. > - merge the sorted file with the db, generating a > <segment,n,inlinks,scores> file. > - sort this file by <segment,n>. <inlinks,scores> comes from the webdb, right? > - merge this with the segment's fetcher/ files, rewriting each with > updated inlinks and score. I'm a bit fuzzy on this one. Where are inlinks and score stored in a segment? FetchListEntry.anchors and FetchList.page.score/nextScore? --Matt On Fri, 10 Dec 2004 11:00:35 -0800, Doug Cutting <[EMAIL PROTECTED]> wrote: > Xin-Yi Liu wrote: > > is there a way to update these anchor texts without > > refetching every single page in your web db? > > Not at present. I hope to soon add a tool that will do this. It would, > given a set of segments, rewrite all of their fetcher/ data to include > the db's current score and inlinks for each url. > > The algorithm (in case someone else wishes to implement it) would be > something like: > - iterate through fetcher/ data, generating a <url,segment,n> output > file, where n is the url's position within the segment. > - sort this file by url. > - merge the sorted file with the db, generating a > <segment,n,inlinks,scores> file. > - sort this file by <segment,n>. > - merge this with the segment's fetcher/ files, rewriting each with > updated inlinks and score. > > SequenceFile would be used for all of the intermediate files and for the > sorting. > > This tool would be great to have! > > Doug > > > ------------------------------------------------------- > SF email is sponsored by - The IT Product Guide > Read honest & candid reviews on hundreds of IT Products from real users. > Discover which products truly live up to the hype. Start reading now. > http://productguide.itmanagersjournal.com/ > _______________________________________________ > Nutch-developers mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/nutch-developers > ------------------------------------------------------- The SF.Net email is sponsored by: Beat the post-holiday blues Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek. It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt _______________________________________________ Nutch-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-developers
