rzo1 opened a new issue, #2094:
URL: https://github.com/apache/stormcrawler/issues/2094
## What happens
`HttpProtocol.configure()` builds the browser context options and, when
`http.proxy` is set, it calls `setIgnoreHTTPSErrors(true)` right after
`setProxy()`. The two calls are tied together: there is no configuration key
that controls the flag, and no way to keep certificate validation on while
fetching through a proxy. The flag is a `NewContextOptions` setting, so it
applies to the whole context: every page, every navigation and every
subresource, not just the leg that terminates at the proxy. Without a proxy the
browser validates normally, so the behaviour flips on an unrelated setting.
## Where
`external/playwright/src/main/java/org/apache/stormcrawler/protocol/playwright/HttpProtocol.java:162-170`,
config keys `http.proxy`, `http.proxy.username`, `http.proxy.password`.
```java
final Proxy globalProxy = getProxy(proxyServer, proxyUser, proxyPwd);
if (globalProxy != null) { b_c_options.setProxy(globalProxy);
b_c_options.setIgnoreHTTPSErrors(true);
}
```
## Why it matters
An operator who configures a proxy gets a browser that renders any HTTPS
page whose certificate does not verify, and the browser executes the JavaScript
on that page. Nothing in `playwright-conf.yaml` or the module README mentions
the behaviour, so the operator has no reason to expect it and no key to turn it
off. Core has a comparable setting for okhttp (`http.trust.everything`,
documented in configuration.adoc), which is exactly what is missing here: a
named, documented switch rather than a side effect. The exposure needs someone
able to intercept or answer for the target host, so it only bites deployments
where the network path between the proxy and the origin is not trusted.
## Reproduction
No automated test. `configure()` launches or connects to a real Chromium
instance, and the flag only shows up in the browser's TLS handshake, so
reproducing it needs a browser and a TLS endpoint. The module's own
`ProtocolTest` already requires a running Chrome for the same reason.
Manual steps:
1. Serve any page over HTTPS with a certificate the container does not trust
(for example a self-signed one).
2. Configure the playwright protocol without a proxy and fetch the URL. The
fetch fails on the certificate.
3. Set `http.proxy` to a proxy that forwards to the same host and fetch
again. The page is fetched and rendered.
## Suggested fix
Introduce a dedicated key, for example `playwright.ignore.https.errors`,
read it in `configure()` and pass it to `setIgnoreHTTPSErrors()` independently
of whether a proxy is set. Default it to false and log a warning when it is
enabled. Document it in `playwright-conf.yaml` next to the other playwright
keys. Note the compatibility impact: deployments running behind a
TLS-intercepting egress proxy currently depend on the implicit behaviour and
will need either the new key or the proxy CA installed in the browser image.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
