Hi StormCrawlers! We have only 2 issues left before we can release SC 4.0:
- https://github.com/apache/stormcrawler/issues/2211 (migrate core to HTTPClient 5) - actually about removing it altogether - https://github.com/apache/stormcrawler/pull/2262 (remove DOM) While work continues on these two items, it would be great if you could run crawls using the code on the main branch - in particular using the OpenSearch module as it has seen some serious refactoring. We have made many changes since the last release, and since the next one is a major release, some of these changes are quite radical. We can always patch issues afterward, but it would be best to anticipate as much as possible. Running the brand new Spout <https://github.com/apache/stormcrawler/pull/2226> and StatusUpdaterBenchmarks <https://github.com/apache/stormcrawler/pull/2213> with various backends is also a good way to test things. The URLFrontier module has also seen changes. If you want to benchmark the backend, make sure you build from source <https://github.com/crawler-commons/url-frontier> as some recent performance improvements have not yet been released. Happy crawling! -- *Julien Nioche * Founder & Director digitalpebble.com <http://www.digitalpebble.com/> Need to have a chat? book a slot here <https://calendar.app.google/vDHtqCpo8G4kNPtR8>
