abhinav-phi commented on PR #2122: URL: https://github.com/apache/stormcrawler/pull/2122#issuecomment-5574558989
Thanks — pushed 18f394a8: - **`URLDecoder` is out:** `getCanonicalHost` now uses a hand-rolled percent-decoder that leaves `+` alone and keeps malformed escapes as literal characters — it never throws, so `http://exa%zz.org/` falls back to the raw spelling instead of blowing up `FetchItem.create`/`getCacheKey` on hostile input. - **Javadoc:** reworded — the method returns "the form used to key politeness queues and the robots.txt cache: what okhttp connects to, with the root label normalised away". - **All three queue modes:** `byIP` and `byDomain` now resolve and split the canonical host too, computed once before the branch in both `FetcherBolt` and `SimpleFetcherBolt`. `http://%65xample.org/` shares a queue with `http://example.org/` in every mode now. - **IDN:** recorded as #2145 with the reproduction, per your suggestion. - The redundant trailing `.toLowerCase(Locale.ROOT)` in the two politeness-key methods is still there on purpose — `key` can be an IP address or a PLD at that point, not the canonical host, so the lowercasing is not redundant for those branches. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
