Sebastian Nagel created NUTCH-3229:
--------------------------------------
Summary: Fetcher: canonicalize host names used for queues and
robots.txt cache
Key: NUTCH-3229
URL: https://issues.apache.org/jira/browse/NUTCH-3229
Project: Nutch
Issue Type: Bug
Components: fetcher, urlnormalizer
Affects Versions: 1.23
Reporter: Sebastian Nagel
Fix For: 1.24
If host names used to manage fetcher queues and the robots.txt are not properly
canonicalized, there may be two queues sending requests to effectively the same
host. That is, the crawl delay is not properly ensured.
For the robots.txt cache: the robots.txt is fetched twice. Not ideal, because
it means an additional request to the host. For the crawler it means to waste
resources needlessly.
See [stormcrawler#2082|https://github.com/apache/stormcrawler/issues/2082] /
[#2122|https://github.com/apache/stormcrawler/pull/2122] which describes the
problem in detail.
Addressing NUTCH-3227 weakens the issue, but it still recommends to
canonicalize the host names. An open question is where the canonicalization
happens: adding a method to the URL normalizer API, or implement it in a
central place (e.g. URLUtil).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
