I wouldn't be so sure that what you think Google does it what it actually does. "Empty" results also denote pages to which the crawler retrieved references (links) from other pages, but didn't have time to fetch the content yet. Google displays the page because it uses contextual terms from links pointing to it (it doesn't require the content of the actual page to be present).
In my opinion what you're trying to achieve (display links to pages that are blocked from the crawler by robots.txt) is a violation of the robots exclusion protocol; after all, if I wanted the crawler to index and display those pages, I wouldn't put them under robots.txt, right?
I'm looking forward to your point of view on the subject.
Dawid
Andrew Chen wrote:
Ah, on second thought, it looks easier than I thought.
Looks like I might be able to return an empty Content object from the http-protocol plugin (in Http.java), comment out the Exception, and it should work... I'll try it out.
It might be worthwhile to make this the default Nutch behavior though, especially if Google handles it this way!
Sorry for the spam ;) Andrew
On Sun, 3 Oct 2004 23:13:39 -0700, Andrew Chen <[EMAIL PROTECTED]> wrote:
Hi everyone,
I'm trying very very hard to not modify the core Nutch code, and build everything as plug-ins. Kudos to Doug, Mike, and everyone else for building a system that's so easily extended. I haven't had to make any core changes except on one occasion, involving the page score thread discussion we had a while back.
So one thing I want to do is that when a page is linked, but is blocked by a robots.txt, I want it to index the link to the page but not the actual text itself. Google does this - it just shows as an entry:
foo.blah.com/ Similar pages
Right now in Nutch, the default behavior is that a ResourceGoneException is thrown in Http.java, and these links don't get indexed.
I can hack around this by making modifications to the http-protocol plug-in, that fakes an HttpResponse that's empty, so that it indexes. There's probably a cleaner way by modifying the core code.
Is there a better way and/or would this a simple change one of the devs working on the next version of Nutch might incorporate?
Andrew
------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
