Riddhish1 commented on code in PR #2116:
URL: https://github.com/apache/stormcrawler/pull/2116#discussion_r3957106591
##########
external/tika/src/main/java/org/apache/stormcrawler/tika/ParserBolt.java:
##########
@@ -162,11 +162,43 @@ public void execute(Tuple tuple) {
// check that the mimetype is in the whitelist
if (!mimeTypeWhiteList.isEmpty()) {
boolean mt_match = false;
- // see if a mimetype was guessed in JSOUPBolt
+ // parse.Content-Type is assumed byte-detected (JSoupParserBolt
uses Tika detection,
+ // not the raw server header). A custom upstream writing a
header-copied value bypasses
+ // this check — that is a caller responsibility.
String mimeType = metadata.getFirstValue("parse.Content-Type");
- // otherwise rely on what could have been obtained from HTTP
if (mimeType == null) {
- mimeType = metadata.getFirstValue(HttpHeaders.CONTENT_TYPE,
this.protocolMDprefix);
+ // parse.Content-Type is absent: detect from content bytes so
that
+ // the whitelist is evaluated against the same type Tika will
use
+ // to select a parser, not the server-declared HTTP header
which is
+ // untrusted and may differ from what the bytes actually are.
+ String httpCTHint =
+ metadata.getFirstValue(HttpHeaders.CONTENT_TYPE,
this.protocolMDprefix);
+ org.apache.tika.metadata.Metadata detectionMd =
+ new org.apache.tika.metadata.Metadata();
+ if (StringUtils.isNotBlank(httpCTHint)) {
+ // pass the header as a hint only — detect() weighs it but
+ // content bytes take precedence
+
detectionMd.set(org.apache.tika.metadata.Metadata.CONTENT_TYPE, httpCTHint);
+ }
+ // pass the filename so detection matches what the parser
dispatches on;
+ // without it, an ambiguous byte sequence (e.g. plain text
with a .html
+ // extension) can resolve differently here than at parse time
+ try {
+ URL _url = URLUtil.toURL(url);
+ detectionMd.set(TikaCoreProperties.RESOURCE_NAME_KEY,
_url.getFile());
+ } catch (MalformedURLException e1) {
+ throw new IllegalStateException("Malformed URL", e1);
+ }
+ try {
+ mimeType = tika.detect(new ByteArrayInputStream(content),
detectionMd);
+ } catch (IOException e) {
+ LOG.warn("Failed to detect MIME type for {}: {}", url,
e.getMessage());
+ }
+ if (mimeType != null) {
+ // write back so downstream code and metadata consumers see
+ // the same value (avoids a second detection pass)
Review Comment:
Fixed clarified that write back only applies to the rejected-tuple path as
AutoDetectParser re-detects on successful parsing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
