BTW, I ran the Fetch with parsing set to true;
----- Original Message -----
From: qi wu
To: [email protected]
Sent: Tuesday, April 03, 2007 12:15 AM
Subject: Fetcher2 too many spinWaiting, How to tune?
Hi,
I am using Fetcher2 with 200 threads started. I get a satisify speed(about
20pages/s) at the beginning stage ,but after no more than one hour,there are
many spinWaiting threads.
"2007-04-02 15:32:07,350 WARN fetcher.Fetcher2 - -activeThreads=199,
spinWaiting=198, fetchQueues.totalSize=10000"
Below are the configurations:
Seed URLs: 2000,Only one URL for one website.
OS: Linux 2.4.21
CPU : Intel(R) Xeon(TM) CPU 3.00GHz *2
Memory: 2G
CPU usage is very low,no more than 50%.Memory usage is very high,only 10%
memory left during fetch,start crawl with "-xmx1000M".The thread dump thread
show they all are waiting at "Thread.sleep(500)".
spinWaiting.incrementAndGet(500);
try {
Thread.sleep(500);
} catch (Exception e) {
}
spinWaiting.decrementAndGet();
Where might be the bottleneck? network, memory or anyplace else? Could you
also give me some hints on how to get more detailed debug info?
Thanks
-Qi
