GitHub user rzo1 added a comment to the discussion: Help us test StormCrawler 4.0 before the release
I ran the [SpoutBenchmark](https://github.com/apache/stormcrawler/pull/2226) and [StatusUpdaterBenchmark](https://github.com/apache/stormcrawler/pull/2213) with the setup from [stormcrawler-benchmark](https://github.com/DigitalPebble/stormcrawler-benchmark) against OpenSearch, URLFrontier and Solr. **Setup** - `main` at 21b1646b (https://github.com/apache/stormcrawler/commit/21b1646b), built from source - Apple M5 Max (18 cores), 128 GB RAM, internal SSD, macOS 26.6 - Docker Desktop 29.9.2, VM with 18 threads and 8 GB RAM, one backend at a time, data inside the container rather than bind-mounted - Java 25.0.5, Storm 3.1.0, utilities run with `storm local`, `-Xmx8g` - 4 GB heap for every backend (6 GB doesn't fit in the VM) - OpenSearch 3.9.0, `crawlercommons/url-frontier:latest` (2.6) with the repo's `config.ini`, Solr 10.1.0 in cloud mode - `benchmark-conf.yaml` from the repo. For Solr I added the `stormcrawler-solr` dependency and: ```yaml solr.status.url: "http://localhost:8983/solr/status"; solr.status.routing.fieldname: "key" solr.status.routing.shards: 10 solr.status.bucket.field: key solr.status.bucket.maxsize: 10 solr.status.max.results: 1000 solr.status.batchUpdateSize: 500 ``` The `status` collection was created with 10 shards and `compositeId` routing on `key`, so that the 10 spouts match the 10 shards, as `SolrSpout` requires. - Input: synthetic, `./generate-input.py --queues 100000 --urls-per-queue 100 --fanout 50`, which gives 10.1M lines and 10M unique URLs on 100k hosts. The figures can't be compared with the ones in the repo, which come from a real crawl dump four times larger. **Writes** (1 status updater) | Backend | Time | URLs / sec | |---|---:|---:| | baseline | 0.8 s | 12,154,031 | | URLFrontier 2.6 | 40 s | 252,386 | | OpenSearch 3.9.0 | 56 s | 180,116 | | Solr 10.1.0 | 2 min 41 s | 62,883 | **Reads** (10 spouts, 1 status updater, 3 min) | Backend / spout | Unique URLs / sec | Unique URLs in 3 min | |---|---:|---:| | URLFrontier 2.6 | 24,973 | 4,495,068 | | OpenSearch, `HybridSpout` | 20,969 | 3,774,598 | | OpenSearch, `AggregationSpout` | 16,040 | 2,887,391 | | Solr, `SolrSpout` | 8,055 | 1,450,008 | No failed URLs in any run, and every backend held exactly 10M URLs after the writes. URLFrontier emitted 3 URLs twice; the other backends emitted no duplicates. A second OpenSearch write run gave 218,984 URLs / sec, so expect some variance between runs. The page cache of the Docker VM was dropped once during the `AggregationSpout` read run, which may have lowered that figure a bit. These are single runs on a laptop. **Found** - `run-benchmark.sh` didn't work on macOS: `/proc`, `drop_caches`, `swapoff`, `nproc`, and decimal commas in the CSVs with a German locale. Fix: DigitalPebble/stormcrawler-benchmark#1 - `SolrSpout` throws an exception unless the number of spouts equals the number of `status` shards. This is by design, but worth keeping in mind when benchmarking Solr with several spouts. GitHub link: https://github.com/apache/stormcrawler/discussions/2268#discussioncomment-19348203 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
