(Would try tonight, but I need to think about how I'll validate the output first...)
You can validate by making calls to the WebDB API. These are too slow to be used to update lots of big segments, but for testing with a few small segments they should work fine.
After a segment has been updated you can open a WebDBReader and iterate through each segment's fetcher data and check that each entry's page now has the same score as WebDBReader.getPage(url).getScore() and that each entry's anchors now match those in WebDBReader.getPage(url).getLinks(url).
Since segments are not sorted by URL, these calls are random-accesses to the DB and thus this would not be a scalable way to implement segment udpate, but it's a fine way to test.
Doug
------------------------------------------------------- The SF.Net email is sponsored by: Beat the post-holiday blues Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek. It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt _______________________________________________ Nutch-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-developers
