jnioche opened a new pull request, #2252:
URL: https://github.com/apache/stormcrawler/pull/2252
Related to #158.
jsoup's `Element.selectXpath()` is not DOM-free: on every call it converts
the whole document to a W3C DOM (`NodeUtils.selectXpath` → `W3CDom.fromJsoup`),
creates a new `XPathFactory` and compiles the expression. `jsoup.XPathFilter`
calls it at least once per key and `jsoup.LinkParseFilter` once per pattern, so
a page was converted once per expression evaluated.
## Changes
- `jsoup.XPathFilter` builds the W3C DOM once per page (`W3CContext`,
namespace-unaware as in jsoup's `selectXpath`) and evaluates all its
expressions against it, mapping the results back to the jsoup elements with
`W3CDom.sourceNodes()`.
- The XPath expressions are compiled once, when the filter is configured.
- `jsoup.LinkParseFilter` uses the same shared context.
The extraction syntax (`/tidyText()`, `/allText()`, `/html()`, `/@attr`) and
the results are unchanged.
## Performance
Old vs new `XPathFilter` + `LinkParseFilter` with the archetype's
`jsoupfilters.json` (6 XPath expressions over 4 keys + 1 link pattern), on the
test pages from `core/src/test/resources`. Average time per page after warm-up,
single thread, JDK 17. The outputs (metadata and outlinks) were checked to be
identical.
| page | size | jsoup parse | W3C conversion | before | after | speedup |
|---|---:|---:|---:|---:|---:|---:|
| stormcrawler.apache.org.html | 7 KB | 0.05 ms | 0.08 ms | 0.69 ms | 0.38
ms | 1.8x |
| stackexception.html | 173 KB | 16.7 ms | 12.9 ms | 75.6 ms | 35.2 ms |
2.1x |
| longtext.html | 5.7 MB (mostly text) | 5.6 ms | 0.10 ms | 0.69 ms | 0.40
ms | 1.7x |
On a page with a large DOM, a single W3C conversion costs about as much as
the jsoup parsing itself; before this change the two filters converted the page
~5 times, taking more than 4x the time spent parsing it. On small pages the
gain comes mostly from no longer creating an `XPathFactory` and compiling the
expressions on every call. The gain grows with the number of expressions
configured.
## Behaviour changes
- An invalid XPath expression now fails when the filter is configured
("Can't compile expression") instead of being logged on every page.
- jsoup's `W3CDom.XPathFactoryProperty` system property (to plug in another
XPath engine) is no longer taken into account; expressions are compiled with
the default JDK `XPathFactory`.
## Not covered
Each filter still does its own conversion, so a configuration with both
`XPathFilter` and `LinkParseFilter` (as in the archetypes) converts the page
twice. Sharing the W3C DOM across filters would require caching it per page,
which could be stale if a previous JSoupFilter modified the document.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
