GitHub user rzo1 added a comment to the discussion: Help us test StormCrawler 4.0 before the release
I ran a couple of short crawls of `main` at 2aa44dc0626ffe75dfd6e3dcedf3df200358a7d4 (https://github.com/apache/stormcrawler/commit/2aa44dc0626ffe75dfd6e3dcedf3df200358a7d4) with the WARC setup from https://github.com/rzo1/sc-warc-demo (branch `sc-4.0-testing`). No long run, I'm on wifi only. **Setup** - `main` at 2aa44dc0, built from source - Storm 3.1.0 (official `storm:3.1.0` images: zookeeper, nimbus, supervisor, ui), URLFrontier 2.6 (`crawlercommons/url-frontier:2.6`, default `-Xms2G -Xmx2G`), see the `docker-compose.yml` in the branch - Topology: URLFrontier spout -> partitioner -> fetcher -> sitemap -> feed -> JSoup parser -> `WARCHdfsBolt`, plus the status updater. Parallelism 1 everywhere. - Apple M5 Max (18 cores), 128 GB RAM, internal SSD, Docker on macOS, Wi-Fi - Java 25 (25.0.4.1 in the Storm image), 1 worker with `-Xmx2g -Djava.net.preferIPv4Stack=true -Djdk.xml.maxGeneralEntitySizeLimit=0 -Djdk.xml.totalEntitySizeLimit=0` - `fetcher.threads.number: 50`, `topology.max.spout.pending: 100`, `urlfrontier.max.buckets: 50`, `urlfrontier.max.urls.per.bucket: 10`, `http.content.limit: 65536`, `fetcher.thread.timeout: 60`, `sitemap.discovery: true`, `http.allow.redirects: true`, okhttp protocol - Seeds: first 1000 hosts of Tranco GQ32K **Results** | Run | Duration | Fetched | Pages/s | |---|---|---|---| | 1 | 7 min | 8,720 | ~21 avg, 30 peak | | 2 | 12 min | 17,483 | 25-28 after ramp-up, 30 peak | Run 2: 14,339 x 200, 1,901 redirects, 428 x 403, 419 x 404, 140 x 429, 102 fetch timeouts, ~690 MB fetched, no ERROR in the worker log. WARC files rotate at 50 MB and are valid after killing the topology. The first minutes are slower (~9-15 pages/s) while robots.txt is fetched for the 1000 hosts. On wifi the network is likely the limit, so these numbers say more about stability than throughput. **Found** - #2282 `SitemapFilter` drops redirect targets on hosts that declare a sitemap in robots.txt. Run 2 logged ~2,000 "Redirect target ... rejected by the URL filters" (not all of them are this case, self-redirects are rejected too). The bolt-level part is already in 3.7.0; with `http.allow.redirects: true` it is new since #2128. Fix: #2285 - #2283 the archetype still used URLFrontier 2.4 - #2284 `WARCRequestRecordFormat` logs every HTTP/2 request line at INFO - The security page on the website still describes the 3.x defaults (`http.trust.everything` on, `file` in `protocols`, no IP filter). `main` is stricter now, so it should be updated with the release. - The archetype sets the `jdk.xml.*` limits to 0 (unlimited). Maybe a large finite value would be better. Not StormCrawler, but maybe useful for others: the WARC flux needs `withConfigKey("warc")` on the bolt, otherwise the `RawLocalFileSystem` config isn't used and the WARC file is truncated when the topology is killed. Also, URLFrontier 2.6 keeps its state in RocksDB inside the container, so `docker restart` doesn't give you an empty frontier. I'll re-run this once the fixes (and Tika 4.2.0) are in, this time on ethernet. GitHub link: https://github.com/apache/stormcrawler/discussions/2268#discussioncomment-19263830 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
