GitHub user rzo1 added a comment to the discussion: Help us test StormCrawler 4.0 before the release
I re-ran the crawl from above on `main` at a9d13bed (https://github.com/apache/stormcrawler/commit/a9d13bed), with #2285. Same setup: [sc-warc-demo](https://github.com/rzo1/sc-warc-demo) (branch `sc-4.0-testing`), the first 1000 hosts of GQ32K, 12 minutes, still on Wi-Fi. | | 29 Oct (2aa44dc0) | 2 Nov (a9d13bed) | |---|---:|---:| | Fetched | 17,483 | 14,268 | | 200 | 14,339 | 12,928 | | 3xx (redirect not followed) | 1,901 | 395 | | "Redirect target ... rejected" | ~2,000 | 491 | | 403 / 404 / 429 | 428 / 419 / 140 | 339 / 420 / 88 | | Fetch timeouts | 102 | 74 | | Fetched | ~690 MB | 634 MB | | Hosts in the WARCs | 534 | 728 | - Redirects are now followed on hosts which declare a sitemap. Fewer fetches end on a 3xx, and more hosts make it into the WARCs. The remaining rejections are targets on another domain (`HostURLFilter` with `ignoreOutsideDomain`), self-redirect loops, and images. None come from `SitemapFilter`. - The worker log has 4,183 `[ERROR]` lines from `SiteMapParserBolt`: sitemaps cut off by `http.content.limit: 65536` that then fail to parse. These were probably in the first run as well. That log line hasn't changed since 2024, and I had grepped for ` ERROR ` while the log writes `[ERROR]`. - WARC files are valid after killing the topology. **Found** - #2299 `SelfURLFilter` drops redirects which only change the case of the path, e.g. `.../ace.html` to `.../ACE.html`. Fix: #2300 GitHub link: https://github.com/apache/stormcrawler/discussions/2268#discussioncomment-19349048 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
