rzo1 commented on code in PR #2169:
URL: https://github.com/apache/stormcrawler/pull/2169#discussion_r4053779464
##########
core/src/main/java/org/apache/stormcrawler/bolt/SiteMapParserBolt.java:
##########
@@ -111,13 +112,16 @@ public class SiteMapParserBolt extends StatusEmitterBolt {
private List<Extension> extensionsToParse;
+ /** Prefix under which the fetcher stores the response headers, see
protocol.md.prefix. */
+ private String protocolMetadataPrefix = "";
+
@Override
public void execute(Tuple tuple) {
Metadata metadata = (Metadata) tuple.getValueByField("metadata");
byte[] content = tuple.getBinaryByField("content");
String url = tuple.getStringByField("url");
- String ct = metadata.getFirstValue(HttpHeaders.CONTENT_TYPE);
+ String ct = metadata.getFirstValue(HttpHeaders.CONTENT_TYPE,
protocolMetadataPrefix);
Review Comment:
Good catch. crawler-commons gets null again and detects the format itself,
the header is only used for the sniff check. Added a test with a gzipped
sitemap declared as `application/xml` with `Content-Encoding: x-gzip`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
