Ah, on second thought, it looks easier than I thought. Looks like I might be able to return an empty Content object from the http-protocol plugin (in Http.java), comment out the Exception, and it should work... I'll try it out.
It might be worthwhile to make this the default Nutch behavior though, especially if Google handles it this way! Sorry for the spam ;) Andrew On Sun, 3 Oct 2004 23:13:39 -0700, Andrew Chen <[EMAIL PROTECTED]> wrote: > Hi everyone, > > I'm trying very very hard to not modify the core Nutch code, and build > everything as plug-ins. Kudos to Doug, Mike, and everyone else for > building a system that's so easily extended. I haven't had to make any > core changes except on one occasion, involving the page score thread > discussion we had a while back. > > So one thing I want to do is that when a page is linked, but is > blocked by a robots.txt, I want it to index the link to the page but > not the actual text itself. Google does this - it just shows as an > entry: > > foo.blah.com/ > Similar pages > > Right now in Nutch, the default behavior is that a > ResourceGoneException is thrown in Http.java, and these links don't > get indexed. > > I can hack around this by making modifications to the http-protocol > plug-in, that fakes an HttpResponse that's empty, so that it indexes. > There's probably a cleaner way by modifying the core code. > > Is there a better way and/or would this a simple change one of the > devs working on the next version of Nutch might incorporate? > > Andrew > ------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
