GitHub user mvolikas added a comment to the discussion: Help us test StormCrawler 4.0 before the release
Here is a first run from my side. #### Setup - Ubuntu 24.04.5 LTS - [Intel Core i78850H](https://www.intel.com/content/www/us/en/products/sku/134899/intel-core-i78850h-processor-9m-cache-up-to-4-30-ghz/specifications.html), 64GB memory, SSD - OpenJDK Runtime Environment (build 25.0.4.1+1-1-24.04.4-Ubuntu) - Apache Maven 4.0.0-rc-7 (6d8f301b38af7399b36b862969d520974944f0b9) - Build from source (commit 5452c1c0a959fe6aa7a180889018795ac596e3e4) - Storm 3.1.0 - Solr 10.0.0 - Everything on the host (no containers) - Solr in single-node mode, one shard per collection, embedded zookeeper #### Configuration Seeds: First 1000 hosts of https://tranco-list.eu/list/GQ32K/1000000 (Full list) Topology: [the Solr archetype default](https://github.com/apache/stormcrawler/blob/main/external/solr/archetype/src/main/resources/archetype-resources/crawler.flux) with parallelism 1 everywhere solr-conf: ```yaml ... solr.status.bucket.field: host solr.status.bucket.maxsize: 250 solr.status.max.results: 2 solr.status.routing.shards: 1 solr.status.batchUpdateSize: 1000 ... ``` crawler-conf: ```yaml ... fetcher.threads.number: 50 fetcher.max.queue.size: 20 topology.max.spout.pending: 1000 ... ``` #### Results status collection: ```json "facet_fields":{ "status":[ "DISCOVERED", 128370, "FETCHED", 8499, "FETCH_ERROR", 1901, "REDIRECTION", 1228, "ERROR", 711 ] }, ``` "docs" collection: ```json "numFound":6735, ``` #### Run ```bash # Start injection and wait for the seeds to be discovered, then stop the job /opt/apache-storm-3.1.0/bin/storm local target/test-1.0-SNAPSHOT.jar org.apache.storm.flux.Flux injection.flux # Start a 10 minute crawl /opt/apache-storm-3.1.0/bin/storm local target/test-1.0-SNAPSHOT.jar org.apache.storm.flux.Flux crawler.flux --local-ttl 600 ``` #### Findings 1. Minor. Wrong option `-c` in the archetype [Solr control script](https://github.com/apache/stormcrawler/blob/main/external/solr/archetype/src/main/resources/archetype-resources/setup-solr.sh#L34). Solr 10.1.0 is out so we can fix this and possible other issues during the dependency update. 2. Some Solr archetype defaults are not appropriate for crawls with many different hosts (e.g. `solr.status.bucket.maxsize`). I've always been testing with the single seed URL `https://apache.org/` but maybe it makes sense to update the archetype with a more realistic scenario. For example with the first 100 URLs of Tranco and properties that contribute to an efficient crawl. I remember having modified the default regex and URL filters so those are candidates for update too. GitHub link: https://github.com/apache/stormcrawler/discussions/2268#discussioncomment-19336551 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
