dpol1 opened a new issue, #2303:
URL: https://github.com/apache/stormcrawler/issues/2303
`FetchItemQueues.finishFetchItem` removes a queue as soon as its last URL is
done. The next URL for that host gets a new queue, which is ready straight
away. If that URL arrives after the queue has emptied, for instance an outlink
of the host's last page, it can be fetched at once, ignoring
`fetcher.server.delay` and the `Crawl-delay` from the robots.txt.
This test fails in `FetchItemQueuesTest`, as a.net/2 is handed out at once:
```java
FetchItemQueues q = queues("fetcher.server.delay", 5.0f);
add(q, "http://a.net/1";);
q.finishFetchItem(q.getFetchItem(), false); // the queue of a.net is removed
add(q, "http://a.net/2";);
Assertions.assertNull(q.getFetchItem());
```
In a local crawl with `fetcher.server.delay: 3`, I got two requests to the
same host 11 ms apart. In 12-minute crawls of GQ32K hosts with the default
delay of 1 s, a fetch started less than 1 s after the previous one to the same
host had ended in 107 of 18,315 cases with OpenSearch, and in 5,422 of 14,151
with URLFrontier, across 799 hosts. The usual gap is just over 1 s. The higher
figure is probably because URLFrontier hands out a host's next URLs sooner than
the OpenSearch spout, so they more often arrive while the delay is still
running. These figures come from the worker logs. I haven't traced them to this
code path, and a shorter `Crawl-delay` could explain some of them.
3.7.0 has the same gap, though not in every case. Its empty queues stay
until a `getFetchItem` scan reaches them, and the threads scan after each fetch
and every 100 ms when idle, so a URL arriving before the scan still waits.
Since #2132 `finishFetchItem` removes the queue itself, so the new queue is
immediately eligible. I ran the 3.7.0 case only on its queue code.
[NUTCH-2947](https://issues.apache.org/jira/browse/NUTCH-2947) fixed the
same problem in Nutch 1.19 by keeping the state of empty queues. I suggest
keeping the time of a host's last fetch until its delay has passed. I can open
a PR with the test and a fix, for 4.0 or later.
cc @GGraziadei, since #2132 expected the per-host rate to stay bounded by
the crawl delay.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
