[
https://issues.apache.org/jira/browse/NUTCH-3227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sebastian Nagel updated NUTCH-3227:
-----------------------------------
Description:
Currently, BasicURLNormalizer does not normalize the host names regarding
IDNA/Punycode or a trailing dot by default. The "defensive" defaults were used
to stay backward-compatible.
However, internally the crawler needs the normalized host names:
# Queues and robots.txt caches may not work as expected if the host name
appears in variants.
# DNS lookups and consequently fetches require the IDNA (ASCII) variant. Even
if the protocol does the conversion internally, normalization may reduce
overhead.
Proposed solution: the following configuration properties should be changed:
- {{urlnormalizer.basic.host.idn}} => {{toAscii}} : normalize to ASCII / IDNA
- {{urlnormalizer.basic.host.idna2008}} => true : use IDNA2008 resp. ([UTS
#46|https://www.unicode.org/reports/tr46/]) as recent web browsers do
- {{urlnormalizer.basic.host.trim-trailing-dot}} => true : the host name
without a trailing dot is most frequently used in URLs.
was:
Currently, BasicURLNormalizer does not normalize the host names regarding
IDNA/Punycode or a trailing dot by default. The "defensive" defaults were used
to stay backward-compatible.
However, internally the crawler needs the normalized host names:
# Queues and robots.txt caches may not work as expected if the host name
appears in variants.
# DNS lookups and consequently fetches require the IDNA (ASCII) variant. Even
if the protocol does the conversion internally, normalization may reduce
overhead.
Proposed solution: the following configuration properties should be changed:
- {{urlnormalizer.basic.host.idn}} => {{toAscii}} : normalize to ASCII / IDNA
- {{urlnormalizer.basic.host.idna2008}} => true : use IDNA2008 resp. ([UTS
#46](https://www.unicode.org/reports/tr46/)) as recent web browsers do
- {{urlnormalizer.basic.host.trim-trailing-dot}} => true : the host name
without a trailing dot is most frequently used in URLs.
> BasicURLNormalizer: change defaults to always normalize host names
> ------------------------------------------------------------------
>
> Key: NUTCH-3227
> URL: https://issues.apache.org/jira/browse/NUTCH-3227
> Project: Nutch
> Issue Type: Improvement
> Components: configuration, urlnormalizer
> Affects Versions: 1.23
> Reporter: Sebastian Nagel
> Priority: Major
> Fix For: 1.24
>
>
> Currently, BasicURLNormalizer does not normalize the host names regarding
> IDNA/Punycode or a trailing dot by default. The "defensive" defaults were
> used to stay backward-compatible.
> However, internally the crawler needs the normalized host names:
> # Queues and robots.txt caches may not work as expected if the host name
> appears in variants.
> # DNS lookups and consequently fetches require the IDNA (ASCII) variant.
> Even if the protocol does the conversion internally, normalization may reduce
> overhead.
> Proposed solution: the following configuration properties should be changed:
> - {{urlnormalizer.basic.host.idn}} => {{toAscii}} : normalize to ASCII / IDNA
> - {{urlnormalizer.basic.host.idna2008}} => true : use IDNA2008 resp. ([UTS
> #46|https://www.unicode.org/reports/tr46/]) as recent web browsers do
> - {{urlnormalizer.basic.host.trim-trailing-dot}} => true : the host name
> without a trailing dot is most frequently used in URLs.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
