BTW, I ran the Fetch with parsing set to true;
----- Original Message -----
From: qi wu
To: [email protected]
Sent: Tuesday, April 03, 2007 12:15 AM
Subject: Fetcher2 too many spinWaiting, How to tune?
Hi,
I am using Fetcher2 with 200 threads started. I get a satisify speed(about
20pages/s) at the beginning stage ,but after no more than one hour,there are
many spinWaiting threads.
"2007-04-02 15:32:07,350 WARN fetcher.Fetcher2 - -activeThreads=199,
spinWaiting=198, fetchQueues.totalSize=10000"
Below are the configurations:
Seed URLs: 2000,Only one URL for one website.
OS: Linux 2.4.21
CPU : Intel(R) Xeon(TM) CPU 3.00GHz *2
Memory: 2G
CPU usage is very low,no more than 50%.Memory usage is very high,only 10%
memory left during fetch,start crawl with "-xmx1000M".The thread dump thread
show they all are waiting at "Thread.sleep(500)".
spinWaiting.incrementAndGet(500);
try {
Thread.sleep(500);
} catch (Exception e) {
}
spinWaiting.decrementAndGet();
Where might be the bottleneck? network, memory or anyplace else? Could you
also give me some hints on how to get more detailed debug info?
Thanks
-Qi
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Nutch-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-general
