dpol1 commented on code in PR #2122:
URL: https://github.com/apache/stormcrawler/pull/2122#discussion_r3992601615
##########
core/src/main/java/org/apache/stormcrawler/util/URLUtil.java:
##########
@@ -253,6 +254,134 @@ public static String getHost(String url) {
}
}
+ /**
+ * Returns the form of the host used to key politeness queues and the
robots.txt cache: what
+ * okhttp connects to, with the root label normalised away. Host strings
which only differ in
+ * escaping or case reach the same server, so both spellings must end up
under one key,
+ * otherwise one server is fetched under several queue ids and its
robots.txt is downloaded once
+ * per spelling.
+ *
+ * @param url The url to check.
+ * @return String The canonical host for the url, or null if the url is
not well formed or has
+ * no host.
+ */
+ public static String getCanonicalHost(URL url) {
+ String host = url.getHost();
+ if (host == null) {
+ return null;
+ }
+ // okhttp percent-decodes the host when it parses the URL; do the same
+ // so keys derived from the URL string agree with what it connects to.
+ // The decoder never throws: crawled content is hostile input, and a
+ // malformed escape falls back to the raw spelling rather than blowing
+ // up the caller
+ String decoded = percentDecodeHost(host);
+ if (decoded.endsWith(".")) {
+ decoded = decoded.substring(0, decoded.length() - 1);
+ }
+ return decoded.toLowerCase(Locale.ROOT);
+ }
+
+ /**
+ * Percent-decodes a host string, leaving {@code +} alone and keeping
malformed escapes as
+ * literal characters. Unlike {@link URLDecoder#decode}, this never throws.
+ */
+ private static String percentDecodeHost(String host) {
+ if (!host.contains("%")) {
+ return host;
+ }
+ StringBuilder sb = new StringBuilder(host.length());
+ for (int i = 0; i < host.length(); i++) {
+ char c = host.charAt(i);
+ if (c == '%' && i + 2 < host.length()) {
+ int hi = Character.digit(host.charAt(i + 1), 16);
+ int lo = Character.digit(host.charAt(i + 2), 16);
+ if (hi != -1 && lo != -1) {
+ sb.append((char) ((hi << 4) | lo));
Review Comment:
Each %XX becomes one char, so `%C3%BC` decodes to `ü` rather than `ü`.
Harmless as a key, but with `status.updater.normalise.hosts` on that spelling
ends up in the stored URL and resolves to a different host. Decode the
collected bytes as UTF-8?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
