AJ, Number of threads 1000 - very good, but... at least existing J2SE from SUN, it will perform ugly!
Preferabe: 32 Processes, 32 Threads each... At least with "The Grinder" and 2 Gb of memory (I don't know about Nutch!)... You should really calculate everything, CPU, memory for each thread, ... 1000 Threads - too much... P.S. Increase "timeout", or decrease "threads". -----Original Message----- From: AJ Chen [mailto:[EMAIL PROTECTED] Sent: Friday, September 23, 2005 1:41 PM To: nutch-user Subject: SocketTimeoutException In my crawling of large number of selected sites, the number of threads is automatically determined by the number of pages on the fetchlist in each fetch/updatedb cycle. Max number of threads is set to 1000. When using 1000 threads, there are lots of SocketTimeoutException in fetching toward the end of the fetch cycle. Any suggestion for reducing SocketTimeoutException? I also notice that the SocketTimeoutException errors are not counted in the error count for segment status. Why is that? Relevant parameters set: http.timeout=10000 http.max.delays=100000 fetcher.threads.fetch= from 10 to 1000 depending on the size of fetchlist Appreciate your help, AJ
