rzo1 opened a new issue, #2097:
URL: https://github.com/apache/stormcrawler/issues/2097
## What happens
`ParserBolt.execute()` builds its text handler as `new
BodyContentHandler(-1)`, which removes the 100000 character limit Tika applies
by default, and there is no configuration key to set one. The resulting text is
materialised as a single `String`, kept in the `ParseData` and emitted in the
tuple alongside the original bytes. The parse itself runs on the executor
thread with no timeout, so a document that takes a long time inside a parser
holds that thread for as long as it takes.
## Where
`external/tika/src/main/java/org/apache/stormcrawler/tika/ParserBolt.java:216`
and `:246`.
```java
ContentHandler textHandler = new BodyContentHandler(-1);
...
tika.getParser().parse(bais, teeHandler, md, parseContext);
text = textHandler.toString();
```
## Why it matters
The only lever an operator has over the size of a parsed document is
`http.content.limit`, which defaults to -1 (no limit) in
`crawler-default.yaml`. A large document therefore reaches the bolt whole and
is held twice, once as bytes and once as text, until the tuple is emitted.
Several fetcher threads doing this at the same time is enough to exhaust a
worker heap, and the tuples get replayed after the restart. The missing timeout
matters less often, but Tika's own guidance for input you do not control is to
parse under one.
A caveat on the obvious reading of this code: Tika does apply a
compression-ratio check of its own. In 3.3.2 that guard sits inside the
container parsers, so a deflated document that expands past the ratio is
aborted before this bolt ever sees the text. That is library behaviour and is
not covered by the test below, and it does nothing for a plain uncompressed
body. What is missing on the StormCrawler side is the character cap and the
timeout.
## Reproduction
Save as
`external/tika/src/test/java/org/apache/stormcrawler/tika/ParserBoltTextLimitTest.java`.
```java
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to you under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.stormcrawler.tika;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.apache.storm.task.OutputCollector;
import org.apache.stormcrawler.Metadata;
import org.apache.stormcrawler.TestUtil;
import org.apache.stormcrawler.parse.ParsingTester;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
/**
* ParserBolt builds its text handler with BodyContentHandler(-1), which
removes the 100000
* character limit Tika applies by default, and there is no configuration
key to put a limit back.
* The whole extracted text is held as one String and emitted in the tuple.
*/
class ParserBoltTextLimitTest extends ParsingTester {
@BeforeEach
void setupParserBolt() {
bolt = new ParserBolt();
setupParserBolt(bolt);
}
@Test
void extractedTextIsBounded() throws IOException {
Map<String, Object> conf = new HashMap<>();
bolt.prepare(conf, TestUtil.getMockedTopologyContext(), new
OutputCollector(output));
byte[] content = "word
".repeat(1_000_000).getBytes(StandardCharsets.UTF_8);
parse("https://example.org/big.txt";, content, new Metadata());
List<List<Object>> emitted = output.getEmitted();
Assertions.assertEquals(1, emitted.size());
int textLength = emitted.get(0).get(3).toString().length();
System.out.println(
"input bytes: " + content.length + " -> extracted
characters: " + textLength);
// Tika's own default for BodyContentHandler is 100000 characters
Assertions.assertTrue(
textLength <= 100_000,
"extracted text should be capped, got " + textLength + "
characters");
}
}
```
Run it:
```
mvn -pl external/tika test -Dtest=ParserBoltTextLimitTest
```
It fails on main and becomes the regression test once a cap exists. It feeds
a 5 MB text body and asserts the extracted text is capped at Tika's own default
of 100000 characters.
```
[INFO] Running org.apache.stormcrawler.tika.ParserBoltTextLimitTest
input bytes: 5000000 -> extracted characters: 5000001
[ERROR] ParserBoltTextLimitTest.extractedTextIsBounded -- Time elapsed:
1.324 s <<< FAILURE!
org.opentest4j.AssertionFailedError: extracted text should be capped, got
5000001 characters ==> expected: <true> but was: <false>
at
org.apache.stormcrawler.tika.ParserBoltTextLimitTest.extractedTextIsBounded(ParserBoltTextLimitTest.java:61)
```
## Suggested fix
Read a limit in `ParserBolt.prepare()`, for example `parser.text.maxlength`,
and pass it to `BodyContentHandler` in `execute()` instead of -1. Pick a
default that is generous but finite and record it in the module documentation.
Add a second key for a parse timeout and run `tika.getParser().parse()` on a
bounded executor so the bolt can fail the tuple instead of holding the thread.
Both changes are visible to operators: documents longer than the cap will be
truncated where they were not before, so the default belongs in the release
notes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
