dpol1 commented on code in PR #2169:
URL: https://github.com/apache/stormcrawler/pull/2169#discussion_r4052757741
##########
core/src/main/java/org/apache/stormcrawler/bolt/SiteMapParserBolt.java:
##########
@@ -111,13 +112,16 @@ public class SiteMapParserBolt extends StatusEmitterBolt {
private List<Extension> extensionsToParse;
+ /** Prefix under which the fetcher stores the response headers, see
protocol.md.prefix. */
+ private String protocolMetadataPrefix = "";
+
@Override
public void execute(Tuple tuple) {
Metadata metadata = (Metadata) tuple.getValueByField("metadata");
byte[] content = tuple.getBinaryByField("content");
String url = tuple.getStringByField("url");
- String ct = metadata.getFirstValue(HttpHeaders.CONTENT_TYPE);
+ String ct = metadata.getFirstValue(HttpHeaders.CONTENT_TYPE,
protocolMetadataPrefix);
Review Comment:
the type now also reaches crawler-commons, which trusts a declared
`application/xml` instead of detecting it: a gzipped sitemap served as xml with
`Content-Encoding: x-gzip` (okhttp only decodes `gzip`) parsed before and fails
now. keep passing null here and use the header for the sniff check only?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
