rzo1 opened a new issue, #2095:
URL: https://github.com/apache/stormcrawler/issues/2095
## What happens
`IPFilterRules` is only used by the okhttp protocol, where it runs as a
network interceptor. The playwright protocol never constructs it, so
`http.filter.ipaddress.include` and `http.filter.ipaddress.exclude` have no
effect on a topology that fetches with playwright. The module already installs
a route handler that sees every request the browser makes, but that handler
only aborts resource types listed in `playwright.skip.resource.types` and
otherwise resumes. This covers the navigation itself, its redirect hops, and
every subresource, XHR, iframe and script request the rendered page issues.
## Where
`external/playwright/src/main/java/org/apache/stormcrawler/protocol/playwright/HttpProtocol.java:243-256`.
```java
page.route( lambdaUrl -> true, route -> { // abort if we know the main page
is a redirection if (status.get() != -1) { ... } else if
(resourceTypesToSkip.contains(route.request().resourceType())) { route.abort();
} else { route.resume(); } });
```
The IP filter itself lives in
`core/src/main/java/org/apache/stormcrawler/protocol/IPFilterRules.java` and is
wired up only in
`core/src/main/java/org/apache/stormcrawler/protocol/okhttp/HttpProtocol.java`.
## Why it matters
This is documented: `crawler-default.yaml:161-162` and
`docs/src/main/asciidoc/configuration.adoc:259-260` both say the IP filter is
okhttp only, and both keys ship commented out, so nobody was told something
untrue. The gap is still worth closing. A crawler that renders pages in a
browser makes far more requests than one that fetches bytes, and every one of
them leaves the worker with no address check, including requests the fetched
page decides to make. An operator who moves a topology from okhttp to
playwright loses the control silently, because the keys stay in the
configuration and stop doing anything.
## Reproduction
No automated test. `configure()` launches or connects to a real Chromium, so
exercising the route handler needs a browser plus an HTTP server bound to a
filtered address.
Manual steps:
1. Configure a topology with the playwright protocol and set
`http.filter.ipaddress.exclude: "localhost,sitelocal"`.
2. Run a small HTTP server on the loopback interface and fetch its URL
through the protocol. It is fetched, whereas the okhttp protocol fails the
fetch with an `IOException`.
3. Serve a page whose body requests a subresource from a loopback URL. The
request is issued by the browser and the route handler resumes it.
## Suggested fix
Build an `IPFilterRules` instance in `HttpProtocol.configure()` from the
same configuration keys, and apply it inside the existing `page.route` handler:
resolve the request host and call `route.abort()` when the address is rejected.
Apply `URLFilters` in the same place so redirect hops and subresource requests
are checked against the crawl scope. Update
`docs/src/main/asciidoc/configuration.adoc` so the "OkHttp only" note reflects
the new state. Resolution in the handler adds a lookup per request, so cache
results per host. A separate DNS lookup is also not the address the browser
ends up connecting to, which is worth stating in the documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
