abhinav-phi opened a new pull request, #2122: URL: https://github.com/apache/stormcrawler/pull/2122
Fixes #2082. okhttp percent-decodes and lowercases the host when it parses the URL, so `http://exampl%65.org/` and `http://example.org/` are one origin at connect time — but `FetchItem.create`, `SimpleFetcherBolt.getPolitenessKey` and `HttpRobotRulesParser.getCacheKey` keyed on the raw host string (only lowercased). Each spelling got its own politeness queue and its own robots.txt cache entry, so the per-host delay was applied several times in parallel to the same server (bounded by `fetcher.threads.number`, not by the number of aliases) and robots.txt was downloaded once per spelling. Adds `URLUtil.getCanonicalHost`, which percent-decodes the host, strips a trailing dot and lowercases it, and uses it at all three call sites. This changes only how URLs are grouped, not which URLs are crawled. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
