rzo1 opened a new issue, #2082: URL: https://github.com/apache/stormcrawler/issues/2082
## What happens `FetchItem.create` uses `u.getHost()` as the politeness queue id and `HttpRobotRulesParser.getCacheKey` builds `protocol:host:port` from the same string. Both only lowercase it. okhttp percent-decodes the host when it parses the URL, so `http://exampl%65.org/` and `http://example.org/` are one origin at connect time but two queue ids and two robots.txt cache entries in the bolt. `BasicURLNormalizer` does not close the gap either: it lowercases the host but leaves percent-escapes in place. ## Where `core/src/main/java/org/apache/stormcrawler/bolt/FetcherBolt.java:189` and `:197` (the same lowercasing at `:170`, and `SimpleFetcherBolt.java:628` and `:631`), `core/src/main/java/org/apache/stormcrawler/protocol/HttpRobotRulesParser.java:69-81`, `core/src/main/java/org/apache/stormcrawler/filtering/basic/BasicURLNormalizer.java:136`. Config keys involved: `fetcher.queue.mode` (default `byHost`), `fetcher.server.delay`, `fetcher.threads.number`. ```java key = u.getHost(); ... queueId = key.toLowerCase(Locale.ROOT); ``` ```java String host = url.getHost().toLowerCase(Locale.ROOT); return protocol + ":" + host + ":" + port; ``` ## Why it matters One server can be addressed under many host spellings that all resolve to the same origin, and each spelling gets its own delay clock and its own robots.txt download. The per-host delay is then applied several times in parallel to the same server, bounded by `fetcher.threads.number` (10 by default), not by the number of aliases. Robots rules are still obeyed, since every alias re-fetches and re-parses the same file, so this is wasted requests and a politeness overrun on the remote side rather than a rules bypass. A crawl that follows outlinks can pick these up from any page that links to the aliased forms. ## Reproduction Save as `core/src/test/java/org/apache/stormcrawler/protocol/HostAliasCacheKeyTest.java`. ```java /* * Licensed to the Apache Software Foundation (ASF) under one or more * contributor license agreements. See the NOTICE file distributed with * this work for additional information regarding copyright ownership. * The ASF licenses this file to you under the Apache License, Version 2.0 * (the "License"); you may not use this file except in compliance with * the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ package org.apache.stormcrawler.protocol; import java.net.URL; import okhttp3.HttpUrl; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; /** * Two URLs whose host strings differ only by percent-escaping or by a trailing dot are sent to the * same server by okhttp, so they must share one robots.txt cache entry. */ class HostAliasCacheKeyTest { @Test void okhttpCollapsesHostAliases() { // what the client actually connects to Assertions.assertEquals( "example.org", HttpUrl.parse("http://%65xample.org/a";).host(), "percent-escaped"); Assertions.assertEquals( "example.org", HttpUrl.parse("http://exampl%65.org/a";).host(), "percent-escaped"); Assertions.assertEquals( "example.org", HttpUrl.parse("http://EXAMPLE.org/a";).host(), "upper case"); } @Test void robotsCacheKeyIsTheSameForHostAliases() throws Exception { String canonical = HttpRobotRulesParser.getCacheKey(new URL("http://example.org/a";)); String escaped = HttpRobotRulesParser.getCacheKey(new URL("http://exampl%65.org/a";)); Assertions.assertEquals(canonical, escaped); } } ``` Run it: ``` mvn -pl core test -Dtest=HostAliasCacheKeyTest ``` `okhttpCollapsesHostAliases` passes and shows what the client connects to; `robotsCacheKeyIsTheSameForHostAliases` fails on main. ``` [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.151 s <<< FAILURE! -- in org.apache.stormcrawler.protocol.HostAliasCacheKeyTest [ERROR] org.apache.stormcrawler.protocol.HostAliasCacheKeyTest.robotsCacheKeyIsTheSameForHostAliases -- Time elapsed: 0.018 s <<< FAILURE! org.opentest4j.AssertionFailedError: expected: <http:example.org:80> but was: <http:exampl%65.org:80> ``` ## Suggested fix Add one host canonicalisation helper in `URLUtil` that percent-decodes the host, lowercases it and strips a trailing dot, and call it from `FetchItem.create`, `SimpleFetcherBolt.getPolitenessKey` and `HttpRobotRulesParser.getCacheKey` so all three agree with what okhttp will connect to. Alternatively reject URLs whose host contains a percent escape in `BasicURLNormalizer.filter`, which is stricter and drops such URLs from the crawl entirely; that changes what an existing crawl accepts, so it needs a release note. The first option is the smaller change and does not alter which URLs are crawled, only how they are grouped. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
