Design a Host table in GORA
---------------------------
Key: NUTCH-882
URL: https://issues.apache.org/jira/browse/NUTCH-882
Project: Nutch
Issue Type: New Feature
Affects Versions: 2.0
Reporter: Julien Nioche
Assignee: Julien Nioche
Fix For: 2.0
Having a separate GORA table for storing information about hosts (and domains?)
would be very useful for :
* customising the behaviour of the fetching on a host basis e.g. number of
threads, min time between threads etc...
* storing stats
* keeping metadata and possibly propagate them to the webpages
* keeping a copy of the robots.txt and possibly use that later to filter the
webtable
* store sitemaps files and update the webtable accordingly
I'll try to come up with a GORA schema for such a host table but any comments
are of course already welcome
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
