Sebastian Nagel created NUTCH-3227:
--------------------------------------
Summary: BasicURLNormalizer: change defaults to always normalize
host names
Key: NUTCH-3227
URL: https://issues.apache.org/jira/browse/NUTCH-3227
Project: Nutch
Issue Type: Improvement
Components: configuration, urlnormalizer
Affects Versions: 1.23
Reporter: Sebastian Nagel
Fix For: 1.24
Currently, BasicURLNormalizer does not normalize the host names regarding
IDNA/Punycode or a trailing dot by default. The "defensive" defaults were used
to stay backward-compatible.
However, internally the crawler needs the normalized host names:
# Queues and robots.txt caches may not work as expected if the host name
appears in variants.
# DNS lookups and consequently fetches require the IDNA (ASCII) variant. Even
if the protocol does the conversion internally, normalization may reduce
overhead.
Proposed solution: the following configuration properties should be changed:
- {{urlnormalizer.basic.host.idn}} => {{toAscii}} : normalize to ASCII / IDNA
- {{urlnormalizer.basic.host.idna2008}} => true : use IDNA2008 resp. ([UTS
#46](https://www.unicode.org/reports/tr46/)) as recent web browsers do
- {{urlnormalizer.basic.host.trim-trailing-dot}} => true : the host name
without a trailing dot is most frequently used in URLs.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
