jnioche opened a new pull request, #2253: URL: https://github.com/apache/stormcrawler/pull/2253
Related to #158. As the next release is a major one, the DOM-based filters that have a JSoupFilter equivalent are removed rather than deprecated. ## Changes - Removed `org.apache.stormcrawler.parse.filter.LDJsonParseFilter` and `org.apache.stormcrawler.parse.filter.LinkParseFilter`. Their JSoupFilter equivalents, `org.apache.stormcrawler.jsoup.LDJsonParseFilter` and `org.apache.stormcrawler.jsoup.LinkParseFilter`, are already used by the archetypes. `jsoup.LDJsonParseFilter` uses a CSS selector and does not need a DOM at all. - `JSONResourceWrapper` (OpenSearch) now delegates `needsDOM()` to the filter it wraps; it used to always return `false`, so a wrapped filter requiring the DOM would get a `null` document. Added `JSONResourceWrapperTest`. - Removed `RobotsTags.extractMetaTags(DocumentFragment)`, which was not called anywhere (`JSoupParserBolt` uses the `String` variant), along with its static XPath expression. - Removed the LD-JSON entry from `test.parsefilters.json` and the corresponding test in `XPathFilterTest`; the JSoup version is covered by `JsoupFilterTest` and `JSoupFiltersTest`. - Docs: removed the two filters from the list of provided ParseFilters and pointed to their JSoupFilter equivalents. ## Migration Users with either filter in `parsefilters.json` need to move it to `jsoupfilters.json` with the `org.apache.stormcrawler.jsoup` class name. The parameters of `LDJsonParseFilter` are unchanged. The XPath expressions of `LinkParseFilter` keep working for attributes (e.g. `//IMG/@src`); element names are lowercased automatically. `parse.filter.XPathFilter` is kept for now: its expression syntax differs from the JSoup version (`string()` and `serialize()`) and it is the only XPath extraction available to the Tika `ParserBolt`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
