rzo1 opened a new issue, #2104:
URL: https://github.com/apache/stormcrawler/issues/2104
## What happens
`ParserBolt.execute()` matches `parser.mimetype.whitelist` against
`parse.Content-Type` and, when that key is absent, against the `Content-Type`
response header. Parsing then calls `tika.getParser().parse(...)`, where the
auto-detect parser picks a parser from the content itself. When the two
disagree, the whitelist decides whether the document is parsed but not which
parser runs. `parse.Content-Type` is written by `JSoupParserBolt` when
`detect.mimetype` is true, so the gap only opens where that key is missing:
`detect.mimetype: false`, or a topology that feeds the Tika bolt without a
JSoup parser in front of it.
## Where
`external/tika/src/main/java/org/apache/stormcrawler/tika/ParserBolt.java:163-183`,
config key `parser.mimetype.whitelist`.
```java
String mimeType = metadata.getFirstValue("parse.Content-Type");
// otherwise rely on what could have been obtained from HTTP
if (mimeType == null) {
mimeType = metadata.getFirstValue(HttpHeaders.CONTENT_TYPE,
this.protocolMDprefix);
}
```
## Why it matters
The whitelist reads as a way of choosing which document types this bolt
handles, and the archetype configuration uses it that way. Where
`parse.Content-Type` is absent, the value being matched is whatever the fetched
server said, so the crawled host decides which side of the gate its bytes land
on, while the parser that runs is chosen from those bytes. The practical effect
is limited: the archetypes do wire `JSoupParserBolt` ahead of Tika with
detection on, and getting further than a mismatched parser needs a defect
inside a Tika parser. Still, a check that does not bind to the parser being
selected is not doing the job it appears to do.
## Reproduction
Save as
`external/tika/src/test/java/org/apache/stormcrawler/tika/ParserBoltWhitelistDetectionTest.java`.
```java
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to you under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.stormcrawler.tika;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.apache.http.HttpHeaders;
import org.apache.storm.task.OutputCollector;
import org.apache.stormcrawler.Constants;
import org.apache.stormcrawler.Metadata;
import org.apache.stormcrawler.TestUtil;
import org.apache.stormcrawler.parse.ParsingTester;
import org.apache.stormcrawler.persistence.Status;
import org.apache.stormcrawler.protocol.ProtocolResponse;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
/**
* When no parse.Content-Type is present, ParserBolt matches
parser.mimetype.whitelist against the
* Content-Type response header, while tika.getParser() dispatches on the
bytes. The two can
* disagree, so the whitelist does not decide which parser runs.
*/
class ParserBoltWhitelistDetectionTest extends ParsingTester {
@BeforeEach
void setupParserBolt() {
bolt = new ParserBolt();
setupParserBolt(bolt);
}
@Test
void whitelistAppliesToTheDetectedType() throws IOException {
Map<String, Object> conf = new HashMap<>();
// the whitelist shipped by the archetypes
conf.put("parser.mimetype.whitelist", "application/.+word.*");
conf.put(ProtocolResponse.PROTOCOL_MD_PREFIX_PARAM, "http.");
bolt.prepare(conf, TestUtil.getMockedTopologyContext(), new
OutputCollector(output));
// no parse.Content-Type: no JSoupParserBolt upstream, or
detect.mimetype disabled
Metadata metadata = new Metadata();
metadata.addValue(
"http." + HttpHeaders.CONTENT_TYPE,
"application/vnd.openxmlformats-officedocument.wordprocessingml.document");
// the body is not a word document
byte[] content = "<html><body><p>not a word
document</p></body></html>"
.getBytes(StandardCharsets.UTF_8);
parse("https://example.org/doc.docx";, content, metadata);
System.out.println("detected type: " +
metadata.getFirstValue("parse.Content-Type"));
System.out.println("emitted documents: " +
output.getEmitted().size());
List<List<Object>> status =
output.getEmitted(Constants.StatusStreamName);
Assertions.assertEquals(
1, status.size(), "content not matching the whitelist should
be rejected");
Assertions.assertEquals(Status.ERROR, status.get(0).get(2));
}
}
```
Run it:
```
mvn -pl external/tika test -Dtest=ParserBoltWhitelistDetectionTest
```
It fails on main. It whitelists the archetype's `application/.+word.*`, sets
the response header to the Word type, supplies an HTML body and no
`parse.Content-Type`, and asserts the document is rejected.
```
[INFO] Running org.apache.stormcrawler.tika.ParserBoltWhitelistDetectionTest
detected type: text/html; charset=ISO-8859-1
emitted documents: 1
[ERROR] ParserBoltWhitelistDetectionTest.whitelistAppliesToTheDetectedType
-- Time elapsed: 1.483 s <<< FAILURE!
org.opentest4j.AssertionFailedError: content not matching the whitelist
should be rejected ==> expected: <1> but was: <0>
at
org.apache.stormcrawler.tika.ParserBoltWhitelistDetectionTest.whitelistAppliesToTheDetectedType(ParserBoltWhitelistDetectionTest.java:73)
```
The detected type is printed from `parse.Content-Type` after the parse, so
the test also shows what the bolt itself ended up dispatching on.
## Suggested fix
In `ParserBolt.execute()`, run Tika detection on the content bytes when
`parse.Content-Type` is absent and match the whitelist against the detected
type rather than the response header. Keep the header only as a detection hint,
which is what it already is at line 204. Detection on bytes costs one extra
pass over the content, and documents whose header and bytes disagree will start
being rejected where they were parsed before, so mention it in the release
notes. Restricting the parser set in `tika-config.xml` is worth documenting
alongside, since that bounds which parsers can be selected at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
