actually realized that there's a bug in RequestScheduler.java wich appears when throttling is not enabled.
attached patch fixes the problem.
br, Sami Siren
hello,
anyone else experiencing this problem that fetcher eats alot (all) of cpu when crawling the web, this might be
ok if crawling is done through huge pipe, but this was happening when fetching through 512 kbit line.
I am running nutch with Redhat 9, jvm build is 1.4.2_04-b05. If anyone is experiencing the same problem I have a
fix that solved it in my case and dropped cpu usage to few percents.
-- Sami Siren
Index: RequestScheduler.java =================================================================== RCS file: /cvsroot/nutch/nutch/src/java/net/nutch/fetcher/RequestScheduler.java,v retrieving revision 1.38 diff -r1.38 RequestScheduler.java 1225c1225 < if ( (nextStats < nextThrottle) || (throttlePeriod <= 0) ) --- > if ( (nextStats < nextThrottle) || (throttlePeriod <= 0) || ( > (throttleMaxBandwidth < 0)))
