http://www.media-style.com/gfx/assets/nutch_character_insg1.gif
Cool!
I would love to re-factor the code to a Extension point and a plugin.
That sounds good to me.
Wouldn't it be sense-full to use the filter in the fetching process? Does the fetcher not follow links directly?
The fetcher does not follow links directly. After a page is fetched, the parser finds its urls and writes them to its output. Then the database updater filters urls before adding them to the db. Finally they're generated for fetch from the db.
This permits one to modify ones filters after fetching and parsing without having to refetch and reparse. So, if you changes filters, you can simply throw your database away and then re-build it by updating it with all of your parser output. Urls which were previously filtered will now be added to the db.
Wouldn't it be useful to have not fetched links in the db as well but just not fetch it?
Perhaps, but database update is a major system bottleneck, so if we add that feature it should be optional, and probably disabled by default.
Isn't a page that have more links to other important pages may be more interesting then just normal pages?
So the google principle reverse with may be a lower weight?
Again, perhaps. There are rumors that Google does a bit of this now. A problem is that this is *really* easy to spam.
For url filtering I have following things in mind.
Filtering based on:
1.) dns whois queries. (http://jakarta.apache.org/commons/net/)
2.) Ip2 geo position database queries (for example http://www.ip2location.com/)
3.) link extraction based on content, for example: keyword - domain thesaurus matching, text classification or ontology.
Any 2 cents?
I think a lot of these could be done as offline computations on the database. For example, one could perform a bunch of whois queries, and then decrease the score of pages whose domain was only recently registered. This way you can easily perform just a single whois lookup per domain, and apply it to all of the urls in the domain.
We should make it easier to add new fields that are stored with each page in the database. We could simply add an attribute-value list, but that could get large and slow database updates considerably. What would be better is to have a registry per database of field names and types, so that field names can be written as single bytes and values can be written untyped. I'll think a bit more about how to implement this...
Doug
------------------------------------------------------- This SF.Net email is sponsored by: GNOME Foundation Hackers Unite! GUADEC: The world's #1 Open Source Desktop Event. GNOME Users and Developers European Conference, 28-30th June in Norway http://2004/guadec.org _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
