GitHub user rzo1 created a discussion: Help us test StormCrawler 4.0 before the release
Hi StormCrawlers! @jnioche suggested this on the dev@ list, and we're posting it here as well to reach a broader audience. We're close to releasing StormCrawler 4.0. The last remaining items, removing the DOM from the parse filters (#2262) and dropping HttpClient from core (#2211), are now done. Before we cut the release, it would be great if you could run crawls with the code on the `main` branch, in particular with the OpenSearch module, which has seen some serious refactoring. We've made a lot of changes since the last release, and since this is a major release, some of them are quite radical. We can always patch issues afterwards, but it's better to catch as much as possible now. Running the new [SpoutBenchmark](https://github.com/apache/stormcrawler/pull/2226) and [StatusUpdaterBenchmark](https://github.com/apache/stormcrawler/pull/2213) against different backends is also a good way to test things. The URLFrontier module has changed as well. If you want to benchmark that backend, build [URLFrontier from source](https://github.com/crawler-commons/url-frontier), as some recent performance improvements haven't been released yet. **Making results comparable** To be able to compare results between setups, please seed your crawls with the same list of URLs: the Tranco top 1M list with the ID [GQ32K](https://tranco-list.eu/list/GQ32K/1000000) (or the first N entries of it), so everyone crawls the same hosts. When you post results here, please include: - the commit of `main` you built from - the backend and its version (OpenSearch, Solr, URLFrontier, ...) - your `docker-compose.yml`, or a description of how the services run - hardware: CPU (model and cores), RAM, disk type, and network if relevant - Java version and the worker JVM options - the relevant parts of your `crawler-conf.yaml` (threads, `topology.max.spout.pending`, batching settings, ...) - what you measured (pages fetched per second, benchmark output, ...) and over how long - anything that broke or looked odd, with logs if possible Happy crawling, and thanks for helping! GitHub link: https://github.com/apache/stormcrawler/discussions/2268 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
