Riddhish1 commented on code in PR #2116:
URL: https://github.com/apache/stormcrawler/pull/2116#discussion_r3902242207
##########
external/tika/src/main/java/org/apache/stormcrawler/tika/ParserBolt.java:
##########
@@ -162,11 +162,32 @@ public void execute(Tuple tuple) {
// check that the mimetype is in the whitelist
if (!mimeTypeWhiteList.isEmpty()) {
boolean mt_match = false;
- // see if a mimetype was guessed in JSOUPBolt
+ // see if a mimetype was detected already (e.g. by JSoupParserBolt)
String mimeType = metadata.getFirstValue("parse.Content-Type");
- // otherwise rely on what could have been obtained from HTTP
if (mimeType == null) {
- mimeType = metadata.getFirstValue(HttpHeaders.CONTENT_TYPE,
this.protocolMDprefix);
+ // parse.Content-Type is absent: detect from content bytes so
that
+ // the whitelist is evaluated against the same type Tika will
use
+ // to select a parser, not the server-declared HTTP header
which is
+ // untrusted and may differ from what the bytes actually are.
+ String httpCTHint =
+ metadata.getFirstValue(HttpHeaders.CONTENT_TYPE,
this.protocolMDprefix);
+ org.apache.tika.metadata.Metadata detectionMd =
+ new org.apache.tika.metadata.Metadata();
+ if (StringUtils.isNotBlank(httpCTHint)) {
+ // pass the header as a hint only — detect() weighs it but
+ // content bytes take precedence
+
detectionMd.set(org.apache.tika.metadata.Metadata.CONTENT_TYPE, httpCTHint);
+ }
+ try {
+ mimeType = tika.detect(new ByteArrayInputStream(content),
detectionMd);
Review Comment:
updated detectionMd to pass TikaCoreProperties.RESOURCE_NAME_KEY from the
URL as well, and added a test case verifying that ambiguous content (like
plain text with a .html URL) resolves consistently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
