Hi everyone, I'm trying very very hard to not modify the core Nutch code, and build everything as plug-ins. Kudos to Doug, Mike, and everyone else for building a system that's so easily extended. I haven't had to make any core changes except on one occasion, involving the page score thread discussion we had a while back.
So one thing I want to do is that when a page is linked, but is blocked by a robots.txt, I want it to index the link to the page but not the actual text itself. Google does this - it just shows as an entry: foo.blah.com/ Similar pages Right now in Nutch, the default behavior is that a ResourceGoneException is thrown in Http.java, and these links don't get indexed. I can hack around this by making modifications to the http-protocol plug-in, that fakes an HttpResponse that's empty, so that it indexes. There's probably a cleaner way by modifying the core code. Is there a better way and/or would this a simple change one of the devs working on the next version of Nutch might incorporate? Andrew ------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
