Sebastian Nagel created NUTCH-3228:
--------------------------------------
Summary: BasicURLNormalizer: add true support for IDNA2008 / UTS
#46
Key: NUTCH-3228
URL: https://issues.apache.org/jira/browse/NUTCH-3228
Project: Nutch
Issue Type: Improvement
Components: urlnormalizer
Affects Versions: 1.23
Reporter: Sebastian Nagel
Fix For: 1.24
Modern web browsers implement true IDNA2008 resp. ([UTS
#46|https://www.unicode.org/reports/tr46/]) to convert between the Unicode and
ASCII form of Internationalized Domain Names. The current "lightweight" support
for IDNA2008 (NUTCH-3176) treats certain domain names not properly (eg.
"straße.de"). See
[crawler-commons#608|https://github.com/crawler-commons/crawler-commons/issues/608]
and
[crawler-commons#609|https://github.com/crawler-commons/crawler-commons/issues/609]
for more context and examples.
One possible solution would be to base Nutch's normalizer on that of
crawler-commons. This would also reduce the maintenance load.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
