rzo1 commented on PR #2183: URL: https://github.com/apache/stormcrawler/pull/2183#issuecomment-5811088871
Thanks @tballison, this works nicely. I pushed a commit on top with a few fixes: - The fork returns the content as metadata (`tk:content`), which ended up in `parse.*` for every document. It's stripped now. - When the fork hit its write limit, the XML came back cut off and the document failed with a `SAXParseException` instead of being trimmed. The handlers now keep what they got before the cut, and the fork stops at `parser.tika.text.maxlength`. - One fork per bolt instance, since `execute()` only parses one document at a time. I removed `parser.tika.pipes.numclients`; the docs also said its default was CPU-based, but it's actually 1. - The fork is started with the worker's own `java.home` instead of whatever `java` is on the PATH. - `log4j-core` and `log4j-slf4j2-impl` are excluded, since the Storm worker provides them. - The HtmlMapper can't be set through `parse-context` either. The fork always uses `DefaultHtmlMapper`, so I updated the docs and the warning. - The archetypes now merge `META-INF/tika/*.idx` and `META-INF/extensions.idx` in the shaded jar. Without that, the Tika bolt fails in any shaded topology because only one `parsers.idx` survives (`tesseract-ocr-parser` is not registered). This one also affects the plain Tika 4 path, not only pipes. I tested it on a Storm 3.1.0 cluster in Docker. Timeouts, an OOM in the fork (reported as `parse crash`, worker unaffected) and a 13MB PDF all behave as expected; the 13MB file went through a temp file without any plugins dir. One thing for the Tika side: with `TimeoutLimits(t, t)` every parse logs "progressTimeoutMillis >= totalTaskTimeoutMillis", and there's no way to switch off the stall detector while keeping a total deadline, since `progress=0` is rejected. Could Tika allow `progress == total`, or treat 0 as off? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
