Interesting, really interesting. I didn't mean to argue about the philosophy, I'm just stunned they do index pages blocked by robots.txt; I think it should not be the case. Maybe it's a mix of our both points -- they index links to all pages (as I suggested based on my gissiped what's-behind-google knowledge), but they don't crawl them if excluded by robots.txt. I wonder what the google guy would have to say about the subject.
Thanks for the info, anyway, I think it is valuable.
D.
Andrew Chen wrote:
Dawid,
Thanks for your e-mail. I hadn't thought of it that way - it still strikes me that one could make an argument that any content on open pages is fair game, even if it is protected by robots.txt somewhere else.
But instead of arguing philosophy, let me give you a Google example, using a rather famous robots.txt...
The text-only version of the First Lady's page is disallowed under: http://www.whitehouse.gov/robots.txt
If you Google: http://www.whitehouse.gov/firstlady/text/
You'll see a link but no text underneath it. If you Google most links, you'll typically see summary text (obviously).
I can give you many more examples of this... the reason I get this impression is that in the past, whenever I see links with no summary text, I sometimes check the robots.txt and have always seen that the blank page there.
Andrew
On Mon, 04 Oct 2004 08:38:17 +0200, Dawid Weiss <[EMAIL PROTECTED]> wrote:
I wouldn't be so sure that what you think Google does it what it actually does. "Empty" results also denote pages to which the crawler retrieved references (links) from other pages, but didn't have time to fetch the content yet. Google displays the page because it uses contextual terms from links pointing to it (it doesn't require the content of the actual page to be present).
In my opinion what you're trying to achieve (display links to pages that are blocked from the crawler by robots.txt) is a violation of the robots exclusion protocol; after all, if I wanted the crawler to index and display those pages, I wouldn't put them under robots.txt, right?
I'm looking forward to your point of view on the subject.
Dawid
Andrew Chen wrote:
Ah, on second thought, it looks easier than I thought.
Looks like I might be able to return an empty Content object from the http-protocol plugin (in Http.java), comment out the Exception, and it should work... I'll try it out.
It might be worthwhile to make this the default Nutch behavior though, especially if Google handles it this way!
Sorry for the spam ;) Andrew
On Sun, 3 Oct 2004 23:13:39 -0700, Andrew Chen <[EMAIL PROTECTED]> wrote:
Hi everyone,
I'm trying very very hard to not modify the core Nutch code, and build everything as plug-ins. Kudos to Doug, Mike, and everyone else for building a system that's so easily extended. I haven't had to make any core changes except on one occasion, involving the page score thread discussion we had a while back.
So one thing I want to do is that when a page is linked, but is blocked by a robots.txt, I want it to index the link to the page but not the actual text itself. Google does this - it just shows as an entry:
foo.blah.com/ Similar pages
Right now in Nutch, the default behavior is that a ResourceGoneException is thrown in Http.java, and these links don't get indexed.
I can hack around this by making modifications to the http-protocol plug-in, that fakes an HttpResponse that's empty, so that it indexes. There's probably a cleaner way by modifying the core code.
Is there a better way and/or would this a simple change one of the devs working on the next version of Nutch might incorporate?
Andrew
------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
