So one thing I want to do is that when a page is linked, but is blocked by a robots.txt, I want it to index the link to the page but not the actual text itself. Google does this - it just shows as an entry:
foo.blah.com/
Similar pages
Right now in Nutch, the default behavior is that a ResourceGoneException is thrown in Http.java, and these links don't get indexed.
I can hack around this by making modifications to the http-protocol plug-in, that fakes an HttpResponse that's empty, so that it indexes. There's probably a cleaner way by modifying the core code.
Perhaps we should add another exception for this case, then handle it in Fetcher.java?
On that note, I don't really like using exceptions for this stuff. Perhaps we should instead have a Content.Status enum:
public Status Content.getStatus();
public class Content.Status {
private String name;
private Status(String name) { this.name = name; }
public static final OK = new Status("O);
public static final RESOURCE_GONE = new Status("RESOURCE_GONE");
public static final BLOCKED = new Status("BLOCKED");
...
}In any case, we could implement it with an exception for now, and make this cleanup later.
Is there a better way and/or would this a simple change one of the devs working on the next version of Nutch might incorporate?
I think this would be a good patch to make.
Note that the boolean FetchListEntry.fetch is to permit pages to be indexed without without fetching content, just anchors & url. The idea is that the FetchListTool could generate high-priority pages with fetch=true, then emit all other pages with fetch=false. Then all the urls in the database could be represented in the index, but only a subset with fetched content.
Pages blocked by robots.txt should get the same treatment, I think.
Doug
------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
