rzo1 opened a new issue, #2282: URL: https://github.com/apache/stormcrawler/issues/2282
With `sitemap.discovery: true` and `SitemapFilter` in the URL filter chain (both are in the archetype), the redirect target of a fetched URL is dropped whenever the site's robots.txt declares a sitemap. The target never reaches the status store. Found while test crawling the Tranco top 1000 for #2268, e.g. ``` HttpProtocol [INFO] Redirect target https://www.intuit.com/ rejected by the URL filters FetcherBolt [INFO] Fetched https://intuit.com/ with status 301 ``` https://www.intuit.com/ was never fetched afterwards. **Cause** `FetcherBolt.discoverSitemaps()` sets `foundSitemap=true` on the metadata of the URL being fetched, before the fetch. The redirect target is then filtered with that metadata, and `SitemapFilter` rejects any URL when `foundSitemap=true` and `isSitemap` is not set. The filter is meant for outlinks of non-sitemap pages, not for redirects. This affects: - the redirect emitted by the fetcher bolt (`http.allow.redirects: false`, the default). This is not new: it is in 3.7.0 too, and the drop is silent. - redirects followed by the okhttp protocol (`http.allow.redirects: true`). In 3.7.0 OkHttp followed them without filters; since #2128 every hop goes through the URL filters, so they are now rejected too. - `SimpleFetcherBolt` (same pattern) and meta refresh redirects in `JSoupParserBolt`, whose tuple metadata carries `foundSitemap` as well. **Tests** I have tests for this and can open a PR with them: `FetcherBolt` / `SimpleFetcherBolt` with a 301 and a robots.txt with and without a `Sitemap:` line, for both values of `http.allow.redirects`. The "with sitemap" cases fail on main, and the bolt-level one fails on 3.7.0 too. There is also a `SitemapFilterTest` (none existed), which passes and covers the outlink behaviour that has to stay. **Possible fix** Don't let `foundSitemap` reach redirect filtering: set it only after the fetch, and filter `REDIRECTION` targets without it, in both fetcher bolts. Outlinks from the parser would still see it, so `SitemapFilter` keeps working as intended. The meta refresh case in `JSoupParserBolt` needs the same treatment. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
