dpol1 commented on code in PR #2116:
URL: https://github.com/apache/stormcrawler/pull/2116#discussion_r3902006382
##########
external/tika/src/main/java/org/apache/stormcrawler/tika/ParserBolt.java:
##########
@@ -162,11 +162,32 @@ public void execute(Tuple tuple) {
// check that the mimetype is in the whitelist
if (!mimeTypeWhiteList.isEmpty()) {
boolean mt_match = false;
- // see if a mimetype was guessed in JSOUPBolt
+ // see if a mimetype was detected already (e.g. by JSoupParserBolt)
String mimeType = metadata.getFirstValue("parse.Content-Type");
Review Comment:
Genuine question: is a pre-existing `parse.Content-Type` trusted by design?
Coming from JSoupParserBolt's own byte detection that seems fine, but any other
upstream writing a server-influenced value there skips the new check entirely.
If it's intentional, worth a comment saying so.
##########
external/tika/src/main/java/org/apache/stormcrawler/tika/ParserBolt.java:
##########
@@ -162,11 +162,32 @@ public void execute(Tuple tuple) {
// check that the mimetype is in the whitelist
if (!mimeTypeWhiteList.isEmpty()) {
boolean mt_match = false;
- // see if a mimetype was guessed in JSOUPBolt
+ // see if a mimetype was detected already (e.g. by JSoupParserBolt)
String mimeType = metadata.getFirstValue("parse.Content-Type");
- // otherwise rely on what could have been obtained from HTTP
if (mimeType == null) {
- mimeType = metadata.getFirstValue(HttpHeaders.CONTENT_TYPE,
this.protocolMDprefix);
+ // parse.Content-Type is absent: detect from content bytes so
that
+ // the whitelist is evaluated against the same type Tika will
use
+ // to select a parser, not the server-declared HTTP header
which is
+ // untrusted and may differ from what the bytes actually are.
+ String httpCTHint =
+ metadata.getFirstValue(HttpHeaders.CONTENT_TYPE,
this.protocolMDprefix);
+ org.apache.tika.metadata.Metadata detectionMd =
+ new org.apache.tika.metadata.Metadata();
+ if (StringUtils.isNotBlank(httpCTHint)) {
+ // pass the header as a hint only — detect() weighs it but
+ // content bytes take precedence
+
detectionMd.set(org.apache.tika.metadata.Metadata.CONTENT_TYPE, httpCTHint);
+ }
+ try {
+ mimeType = tika.detect(new ByteArrayInputStream(content),
detectionMd);
Review Comment:
The parse-time detection further down also gets the filename via
`RESOURCE_NAME_KEY`, this one doesn't — so the two can still disagree (I could
reproduce it with plain-text bytes and a `.html` URL: `text/plain` here,
`text/html` at dispatch). Passing the same filename hint into `detectionMd`
would make the check genuinely match what the parser dispatches on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
