Well, running it with 200 fetcher threads and no delay works for about 20 minutes.. then the file server crashed .
So.... I think that the DNS queries are the issue. I am not able to setup my own DNS server, but I did find this setting in java.security: networkaddress.cache.ttl. Since I am using Java 1.6 it was only caching the entries for 30 seconds. I hope that setting this to -1 will help with all the unecessary calls to DNS. I don't even really need to use DNS... the Ip address and host names of our servers do not change (and if they do I know about it before hand). -- View this message in context: http://lucene.472066.n3.nabble.com/Optimizing-crawling-for-small-number-of-domains-sites-aka-intranet-crawling-tp3804830p3818986.html Sent from the Nutch - User mailing list archive at Nabble.com.
