abhinav-phi commented on PR #2122: URL: https://github.com/apache/stormcrawler/pull/2122#issuecomment-5641274398
Thanks @dpol1 — pushed 57f76f7c: **UTF-8 decoding:** `percentDecodeHost` now collects the decoded octets and interprets them as UTF-8, so `%C3%BC` becomes `ü` rather than the two-char mojibake. Pinned by `multiByteEscapeDecodesAsUtf8`. You're right that it matters beyond the key: with `status.updater.normalise.hosts` on, the decoded form is what lands in the stored URL, so decoding it as UTF-8 keeps it resolvable to the same host the browser would reach. **Partition-key release note:** agreed, added to the PR description's release-note section — enabling `status.updater.normalise.hosts` changes the partition key of any stored URL whose host carried a trailing dot or a `%XX` escape, so in OpenSearch (routing) and URLFrontier (queue) the next update for such a URL lands under the normalised key next to the old record rather than on it. Rare, bounded, and only when the flag is turned on. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
