rzo1 opened a new pull request, #2171: URL: https://github.com/apache/stormcrawler/pull/2171
The protocols copy every response header into the response metadata, and a header name is not restricted to anything, so a fetched server could write the keys the crawler uses to record its own fetch. `WARCRecordFormat` promotes a record from `resource` to `response` whenever `_response.headers_` is non-blank and embeds the block verbatim, and `WARCRequestRecordFormat` writes `_request.headers_` as what the crawler supposedly sent — so the archive could be made to contain a request and response of the server's choosing, with its own `WARC-IP-Address` and `WARC-Date`. Values under `metrics.*` reached the fetcher metric registry the same way, which is the other half of #2099. `ProtocolResponse.isReservedMetadataKey` names the keys the crawler writes itself, and the okhttp and Playwright protocols skip a header which matches. Two details worth a look in review: - In okhttp the keys `HTTPHeadersInterceptor` passes through the response headers stay readable, but only while that interceptor is installed, which is what `http.store.headers` does. Without that condition the change would break `http.store.headers: true`. - The interceptor sets the cipher suite only when there is a handshake, so over `http://` a value from the server survived even with headers stored. It now drops that header in the else branch. This also removes an unguarded base64 decode: a value under `_request.headers_` which was not valid base64 threw `IllegalArgumentException` out of `getProtocolOutput` and failed the fetch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
