Re: Maximum links limit per domain

[Dennis Kubes]

The db.max.outlinks.per.page defines how many links are grabbed from a 
given url.  There isn't a way to grab by domain, including subs, without 
writing a custom tool.

I am almost finished with a new scoring framework that creates a 
webgraphdb.  Part of that process is storing outlinks.  It stores only 
the url though, not the anchor text.  It would be fairly simple to write 
a tool that takes that output and processes for domains and subdomains. 
 Let me know if you want a current patch and I will send it to you.

Dennis

brainstorm wrote:
Hi,

Is there a configurable setting to limit the number of links to fetch
for every domain in nutch ? I'm not referring to the topN setting
which sorts a fetchlist using lucene (or nutch) scoring mechanism.

In other words, I just want to fetch 1500 links from every
"whatever.com" domain *including* all their subdomains.

Ex:

upc.edu : 1500 links hard limit counter
escert.upc.edu: *new* 1500 links hard limit counter
ac.upc.edu: *new* 1500 links hard limit counter

*or* shared links hard limit per domain:

*.upc.edu: 1500 links hard limit

Thanks in advance !