omarbounawarapy commented on issue #2145: URL: https://github.com/apache/stormcrawler/issues/2145#issuecomment-6220919848
Thanks @abhinav-phi. I added the `xn--exampl-gva.org` case and it comes back unchanged. Running the core suite turned up one failure I'd like your call on before I open the PR. `normaliseHost` rewrites the host with `getCanonicalHost`, so it now returns punycode, and `URLUtilNormaliseHostTest.multiByteEscapeDecodesAsUtf8` fails: ``` expected: <http://ünicode.example.org/> but was: <http://xn--nicode-2ya.example.org/> ``` I see two ways to handle it: 1. Apply punycode only to the politeness and robots.txt keys, and keep `normaliseHost` returning the decoded Unicode host. The existing test stays. Rewriting the stored URL to punycode is left to `BasicURLNormalizer`'s `hostIDNtoASCII`, which the archetypes already turn on. 2. Let `normaliseHost` return punycode too and update that test. Both spellings then become one record even with `hostIDNtoASCII` off, but the status updater would rewrite IDN hosts whatever the normalizer is set to, and crawls that already have `status.updater.normalise.hosts` on would get new `xn--` records next to the stored Unicode ones. I'd go with 1, since URL rewriting already happens in the normalizer. Which would you prefer? One more thing: `URLPartitioner` also goes through `getCanonicalHost`, so with OpenSearch status routing on, a URL already stored under a Unicode host gets a new routing key. I'll mention it in the PR either way. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
