jnioche opened a new pull request, #2256: URL: https://github.com/apache/stormcrawler/pull/2256
Relates to #158: a first step towards parse filters that no longer deal with the W3C DOM. With this, JSoupFilters work on everything Tika parses, not only on what the `JSoupParserBolt` handles. ### Changes - **`JSoupBuilder`** (new, in `external/tika`): a SAX handler which builds a JSoup `Document` from the XHTML events produced by Tika, like `DOMBuilder` does for the W3C DOM. It is teed with the link and text handlers, so the document is still parsed once, and it also receives the content replayed from a Tika Pipes fork. Element names are lowercased, adjacent text is merged into a single node, script/style content is kept as data, and the document keeps what it received if the parse stops early. - **`ParserBolt`** loads the JSoupFilters from `jsoup.filters.config.file`, the same key as `JSoupParserBolt`, and runs them before the parse filters. The JSoup document is only built when at least one JSoupFilter is configured (new `JSoupFilters.isEmpty()`). - **Tika Pipes uses the `IdentityHtmlMapper`**: Tika only sends the `ParseContext` components listed in `META-INF/tika/*.idx` to the forked JVM, so the fork used to fall back to `DefaultHtmlMapper`, which drops elements such as `span` and most attributes. The module now registers `IdentityHtmlMapper` in its own `META-INF/tika/parse-context.idx` and passes it with each parse when it is the configured mapper (the default). Shaded jars must merge that file, which the archetype already does. - README: new "JSoup filters" section and updated note on the HtmlMapper under `parser.tika.timeout`. ### Things to be aware of - The filters see Tika's XHTML, not the original content: a PDF becomes `<div class="page">` / `<p>` elements, and Tika rebuilds the `<head>` of an HTML page from its metadata (e.g. `<meta name="dc:subject">` instead of `<meta name="keywords">`). This was already the case for the DOM given to the parse filters. - Deprecating the DOM argument of `ParseFilter` (and removing `DOMBuilder`, `DocumentFragmentBuilder` and xerces) is left for a separate PR. ### Tests - `JSoupBuilderTest`: structure, text merging, script data, base URI, truncated input. - `ParserBoltJSoupFiltersTest`: core's XPath test page and a generated PDF, both in-process and through a Tika Pipes fork, plus no filters configured. The Pipes HTML test fails without the `IdentityHtmlMapper` change. - Existing `ParserBoltTest` and `ParserBoltPipesTimeoutTest` pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
