Hello, I am using Nutch 1.5.1 and I am looking to do something specific with it. I have a few million base domains in a Solr index, so for example: http://www.nutch.org, http://www.apache.org, http://www.whatever.com etc. I am trying to crawl each of these base domains in deploy mode and retrieve all of their sub-urls associated with that domain in the most efficient way possible. To give you an example of the workflow I am trying to achieve: (1) Grab a base domain, let's say http://www.nutch.org (2) Crawl the base domain for all URLs in that domain, let's say http://www.nutch.org/page1, http://www.nutch.org/page2, http://www.nutch.org/page3, etc. etc. (3) store these results somewhere (perhaps another Solr instance) and (4) move on to the next base domain in my Solr index and repeat the process. Essentially just trying to grab all links associated with a page and then move on to the next page.
The part I am having trouble with is ensuring that this workflow is efficient. The only way I can think to do this would be: (1) Grab a base domain from Solr from my shell script (simple enough) (2) Add an entry to regex-urlfilter with the domain I am looking to restrict the crawl to, in the example above that would be an entry that says to only keep sub-pages of http://www.nutch.org/ (3) Recreate the Nutch job file (~25 sec.) (4) Start the crawl for pages associated with a domain and do the indexing My issue is with step #3, AFAIK if I want to restrict a crawl to a specific domain I have to change regex-urlfilter and reload the job file. This is a pretty significant problem, since adding 25 seconds every single time I start a new base domain is going to add way too many seconds to my workflow (25 sec x a few million = way too much time). Finally the question...is there a way to add url filters on the fly when I start a crawl and/or restrict a crawl to a particular domain on the fly. OR can you think of a decent solution to the problem/am I missing something?
