fetcher.server.delay configuration not working

Andrés Rincón Pacheco Fri, 20 Nov 2015 15:36:43 -0800

Hi,

I configured the fetcher.server.delay property with 2.5 as value, but when
nutch is fetching urls, the time fetching between urls is bigger that value
configured.


I attach some information of execution.

Date fetching Difference in seconds
2015-11-11 20:56:49,967 5
2015-11-11 20:56:54,746 5
2015-11-11 20:56:59,391 4
2015-11-11 20:57:04,264 5
2015-11-11 20:57:09,212 5
2015-11-11 20:57:13,873 5
2015-11-11 20:57:18,549 5

and some lines of log.

2015-11-11 20:56:21,674 INFO  fetcher.Fetcher - -activeThreads=50,
spinWaiting=49, fetchQueues.totalSize=359, fetchQueues.getQueueCount=1
2015-11-11 20:56:22,674 INFO  fetcher.Fetcher - -activeThreads=50,
spinWaiting=49, fetchQueues.totalSize=359, fetchQueues.getQueueCount=1
2015-11-11 20:56:23,675 INFO  fetcher.Fetcher - -activeThreads=50,
spinWaiting=50, fetchQueues.totalSize=359, fetchQueues.getQueueCount=1
2015-11-11 20:56:24,676 INFO  fetcher.Fetcher - -activeThreads=50,
spinWaiting=50, fetchQueues.totalSize=359, fetchQueues.getQueueCount=1
2015-11-11 20:56:25,677 INFO  fetcher.Fetcher - -activeThreads=50,
spinWaiting=50, fetchQueues.totalSize=359, fetchQueues.getQueueCount=1
2015-11-11 20:56:25,981 INFO  fetcher.Fetcher - fetching http://www.abcderer
(queue crawl delay=2500ms)
2015-11-11 20:56:26,677 INFO  fetcher.Fetcher - -activeThreads=50,
spinWaiting=49, fetchQueues.totalSize=358, fetchQueues.getQueueCount=1
2015-11-11 20:56:27,678 INFO  fetcher.Fetcher - -activeThreads=50,
spinWaiting=49, fetchQueues.totalSize=358, fetchQueues.getQueueCount=1
2015-11-11 20:56:28,679 INFO  fetcher.Fetcher - -activeThreads=50,
spinWaiting=50, fetchQueues.totalSize=358, fetchQueues.getQueueCount=1
2015-11-11 20:56:29,679 INFO  fetcher.Fetcher - -activeThreads=50,
spinWaiting=50, fetchQueues.totalSize=358, fetchQueues.getQueueCount=1
2015-11-11 20:56:30,680 INFO  fetcher.Fetcher - -activeThreads=50,
spinWaiting=50, fetchQueues.totalSize=358, fetchQueues.getQueueCount=1
2015-11-11 20:56:30,791 INFO  fetcher.Fetcher - fetching http://www.abcderer
(queue crawl delay=2500ms)
2015-11-11 20:56:31,681 INFO  fetcher.Fetcher - -activeThreads=50,
spinWaiting=49, fetchQueues.totalSize=357, fetchQueues.getQueueCount=1
2015-11-11 20:56:32,681 INFO  fetcher.Fetcher - -activeThreads=50,
spinWaiting=49, fetchQueues.totalSize=357, fetchQueues.getQueueCount=1
2015-11-11 20:56:33,682 INFO  fetcher.Fetcher - -activeThreads=50,
spinWaiting=50, fetchQueues.totalSize=357, fetchQueues.getQueueCount=1
2015-11-11 20:56:34,683 INFO  fetcher.Fetcher - -activeThreads=50,
spinWaiting=50, fetchQueues.totalSize=357, fetchQueues.getQueueCount=1
2015-11-11 20:56:35,684 INFO  fetcher.Fetcher - -activeThreads=50,
spinWaiting=50, fetchQueues.totalSize=357, fetchQueues.getQueueCount=1
2015-11-11 20:56:35,960 INFO  fetcher.Fetcher - fetching http://www.abcderer
(queue crawl delay=2500ms)
2015-11-11 20:56:36,684 INFO  fetcher.Fetcher - -activeThreads=50,
spinWaiting=49, fetchQueues.totalSize=356, fetchQueues.getQueueCount=1
2015-11-11 20:56:37,685 INFO  fetcher.Fetcher - -activeThreads=50,
spinWaiting=49, fetchQueues.totalSize=356, fetchQueues.getQueueCount=1
2015-11-11 20:56:38,686 INFO  fetcher.Fetcher - -activeThreads=50,
spinWaiting=50, fetchQueues.totalSize=356, fetchQueues.getQueueCount=1
2015-11-11 20:56:39,687 INFO  fetcher.Fetcher - -activeThreads=50,
spinWaiting=50, fetchQueues.totalSize=356, fetchQueues.getQueueCount=1
2015-11-11 20:56:40,552 INFO  fetcher.Fetcher - fetching http://www.abcderer
(queue crawl delay=2500ms)
2015-11-11 20:56:40,688 INFO  fetcher.Fetcher - -activeThreads=50,
spinWaiting=49, fetchQueues.totalSize=355, fetchQueues.getQueueCount=1
2015-11-11 20:56:41,689 INFO  fetcher.Fetcher - -activeThreads=50,
spinWaiting=49, fetchQueues.totalSize=355, fetchQueues.getQueueCount=1
2015-11-11 20:56:42,690 INFO  fetcher.Fetcher - -activeThreads=50,
spinWaiting=50, fetchQueues.totalSize=355, fetchQueues.getQueueCount=1
2015-11-11 20:56:43,691 INFO  fetcher.Fetcher - -activeThreads=50,
spinWaiting=50, fetchQueues.totalSize=355, fetchQueues.getQueueCount=1
2015-11-11 20:56:44,691 INFO  fetcher.Fetcher - -activeThreads=50,
spinWaiting=50, fetchQueues.totalSize=355, fetchQueues.getQueueCount=1
2015-11-11 20:56:45,128 INFO  fetcher.Fetcher - fetching http://www.abcderer
(queue crawl delay=2500ms)
2015-11-11 20:56:45,692 INFO  fetcher.Fetcher - -activeThreads=50,
spinWaiting=49, fetchQueues.totalSize=354, fetchQueues.getQueueCount=1
2015-11-11 20:56:46,693 INFO  fetcher.Fetcher - -activeThreads=50,
spinWaiting=49, fetchQueues.totalSize=354, fetchQueues.getQueueCount=1
2015-11-11 20:56:47,694 INFO  fetcher.Fetcher - -activeThreads=50,
spinWaiting=50, fetchQueues.totalSize=354, fetchQueues.getQueueCount=1
2015-11-11 20:56:48,695 INFO  fetcher.Fetcher - -activeThreads=50,
spinWaiting=50, fetchQueues.totalSize=354, fetchQueues.getQueueCount=1
2015-11-11 20:56:49,696 INFO  fetcher.Fetcher - -activeThreads=50,
spinWaiting=50, fetchQueues.totalSize=354, fetchQueues.getQueueCount=1
2015-11-11 20:56:49,967 INFO  fetcher.Fetcher - fetching http://www.abcderer
(queue crawl delay=2500ms)
2015-11-11 20:56:50,697 INFO  fetcher.Fetcher - -activeThreads=50,
spinWaiting=49, fetchQueues.totalSize=353, fetchQueues.getQueueCount=1
2015-11-11 20:56:51,698 INFO  fetcher.Fetcher - -activeThreads=50,
spinWaiting=49, fetchQueues.totalSize=353, fetchQueues.getQueueCount=1
2015-11-11 20:56:52,699 INFO  fetcher.Fetcher - -activeThreads=50,
spinWaiting=50, fetchQueues.totalSize=353, fetchQueues.getQueueCount=1
2015-11-11 20:56:53,700 INFO  fetcher.Fetcher - -activeThreads=50,
spinWaiting=50, fetchQueues.totalSize=353, fetchQueues.getQueueCount=1
2015-11-11 20:56:54,701 INFO  fetcher.Fetcher - -activeThreads=50,
spinWaiting=50, fetchQueues.totalSize=353, fetchQueues.getQueueCount=1
2015-11-11 20:56:54,746 INFO  fetcher.Fetcher - fetching http://www.abcderer
(queue crawl delay=2500ms)

What is the wrong in the configuration?

Thanks.

fetcher.server.delay configuration not working

Reply via email to