jnioche opened a new pull request, #2262: URL: https://github.com/apache/stormcrawler/pull/2262
Follow-up to #2252, #2253 and #2256 for #158. Building a W3C DOM is costly, and since the JSoupFilters are now applied by both the `JSoupParserBolt` and the Tika `ParserBolt`, nothing in StormCrawler needs it any more. This removes the DOM from the parse filtering altogether. ### Changes - **API**: `ParseFilter.filter(String url, byte[] content, ParseResult parse)` no longer takes a `DocumentFragment`; `needsDOM()` is removed. `ParseFilters` no longer skips filters for lack of a DOM. - **Removed**: the DOM-based `parse.filter.XPathFilter` (use `jsoup.XPathFilter`) and `DebugParseFilter`. - **Parser bolts**: `JSoupParserBolt` and the Tika `ParserBolt` no longer build a DOM. `DocumentFragmentBuilder` (core) and `DOMBuilder` / `XMLCharacterRecognizer` (Tika) are removed, along with the Tika option `parser.uppercase.element.names`, which only applied to the DOM. - **Dependencies**: `xercesImpl` is dropped from core. - Updated all `ParseFilter` implementations (core, langid, playwright, opensearch `JSONResourceWrapper`), the tests and the documentation. ### Breaking changes (for the 4.0 migration notes) - Custom `ParseFilter`s must drop the `DocumentFragment` argument and any `needsDOM()` override. Filters which extracted data from the DOM should be rewritten as `JSoupFilter`s. - Configurations using `org.apache.stormcrawler.parse.filter.XPathFilter` must move to `org.apache.stormcrawler.jsoup.XPathFilter` in `jsoupfilters.json`. The expression syntax differs: `string(...)` / `serialize(...)` become `/tidyText()`, `/allText()`, `/html()` or `/@attr`. - `parser.uppercase.element.names` is no longer used. ### Testing Core parse / bolt / jsoup tests, all Tika tests, `JsRenderingDetectorTest` and `DelegateRefresherTest` pass locally. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
