rzo1 opened a new issue, #2088:
URL: https://github.com/apache/stormcrawler/issues/2088
## What happens
`HttpProtocol.configure()` passes `http.allow.redirects` to
`OkHttpClient.Builder.followRedirects()`. When it is enabled, okhttp follows
the redirect chain inside the client and the crawler only ever sees the final
response, returned under the original URL. None of the intermediate targets
goes through `URLFilters`, so the scheme exclusions, host and domain
confinement and depth rules configured for the crawl do not apply to them. This
is stated in the documentation (`configuration.adoc:233` and
`internals.adoc:388`) and the setting is off by default. The `IPFilterRules`
interceptor is a network interceptor, so it does run on every hop when it is
configured.
## Where
`core/src/main/java/org/apache/stormcrawler/protocol/okhttp/HttpProtocol.java:161`
on main. Config key: `http.allow.redirects`.
```java
.followRedirects(ConfUtils.getBoolean(conf, "http.allow.redirects", false))
```
## Why it matters
An operator who turns the setting on to keep redirect chains out of the
status store also turns off URL policy for those hops, and the two are
unrelated concerns. A crawled page can then steer a fetch to a URL the filter
chain would have rejected, and the response is stored against the original URL.
The documentation says the targets are not filtered, so nobody is misled, but
there is no way to keep immediate redirects and URL filtering at the same time.
## Reproduction
Save as
`core/src/test/java/org/apache/stormcrawler/protocol/OkHttpFollowRedirectsTest.java`:
```java
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to you under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.stormcrawler.protocol;
import jakarta.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.io.OutputStream;
import java.nio.charset.StandardCharsets;
import org.apache.storm.Config;
import org.apache.stormcrawler.Metadata;
import org.eclipse.jetty.server.Handler;
import org.eclipse.jetty.server.Request;
import org.eclipse.jetty.server.handler.AbstractHandler;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
/** Shows what happens to the redirect target when http.allow.redirects is
on. */
class OkHttpFollowRedirectsTest extends AbstractProtocolTest {
@Override
protected Handler[] getHandlers() {
return new Handler[] {
new AbstractHandler() {
@Override
public void handle(
String target,
Request baseRequest,
jakarta.servlet.http.HttpServletRequest request,
HttpServletResponse response)
throws IOException {
baseRequest.setHandled(true);
if (target.equals("/start")) {
response.setStatus(HttpServletResponse.SC_MOVED_TEMPORARILY);
response.setHeader(
"Location", "http://127.0.0.1:"; + HTTP_PORT
+ "/elsewhere");
response.setContentLength(0);
response.getOutputStream().close();
return;
}
response.setStatus(HttpServletResponse.SC_OK);
response.setContentType("text/plain");
final byte[] content = ("body of " +
target).getBytes(StandardCharsets.UTF_8);
response.setContentLength(content.length);
try (OutputStream out = response.getOutputStream()) {
out.write(content);
}
}
}
};
}
@Test
void redirectTargetIsFetchedAndAttributedToTheSourceUrl() throws
Exception {
Config conf = new Config();
conf.put("http.agent.name", "this_is_only_a_test");
conf.put("http.allow.redirects", true);
org.apache.stormcrawler.protocol.okhttp.HttpProtocol protocol =
new org.apache.stormcrawler.protocol.okhttp.HttpProtocol();
protocol.configure(conf);
ProtocolResponse response =
protocol.getProtocolOutput(
"http://localhost:"; + HTTP_PORT + "/start", new
Metadata());
protocol.cleanup();
// documents the current behaviour: the body of the redirect target
is
// returned under the source URL, and nothing in the response says
which
// URL it came from. Each hop should be run through the URL filters.
Assertions.assertEquals(200, response.getStatusCode());
Assertions.assertEquals("body of /elsewhere", new
String(response.getContent(), "UTF-8"));
Assertions.assertNull(response.getMetadata().getFirstValue("_redirTo"));
}
}
```
Run it:
```
mvn -pl core test -Dtest=OkHttpFollowRedirectsTest
```
It serves a redirect from one origin to another on a local Jetty server. The
test passes on main and documents the present behaviour, since the protocol has
no filter chain to assert against yet.
```
[INFO] Running org.apache.stormcrawler.protocol.OkHttpFollowRedirectsTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.514
s -- in org.apache.stormcrawler.protocol.OkHttpFollowRedirectsTest
```
The assertions are that the body of the redirect target comes back with
status 200 under the source URL and that the response metadata does not record
which URL it came from.
## Suggested fix
Give `HttpProtocol` access to the configured `URLFilters` and add an okhttp
interceptor that runs each redirect target through them before the hop is
taken, aborting the call when a target is rejected. Where wiring the full chain
into the protocol is not wanted, apply at least the scheme allowlist and
`IPFilterRules` per hop and document the remainder. Record the final URL in the
response metadata as well, so callers can tell that the content is not from the
URL they asked for.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
