rzo1 opened a new pull request, #2285: URL: https://github.com/apache/stormcrawler/pull/2285
`FetcherBolt.discoverSitemaps()` sets `foundSitemap=true` on the metadata of the URL being fetched, before the fetch. Redirect targets were then filtered with that metadata, and `SitemapFilter` rejects every URL when `foundSitemap=true` and `isSitemap` is not set. As a result, the redirect target of any URL on a site whose robots.txt declares a sitemap never reached the status store. Changes: - New `StatusEmitterBolt.redirectMetadata(Metadata)`, which returns a copy of the metadata without `foundSitemap` if it is present. - It is used for the metadata passed to `protocol.getProtocolOutput` in `FetcherBolt` and `SimpleFetcherBolt` (okhttp filters each redirect hop with it since #2128), for the `REDIRECTION` outlinks emitted by both fetcher bolts, and for meta refresh redirects in `JSoupParserBolt`. - The parse and status tuples still carry `foundSitemap`, so `SitemapFilter` still applies to the page's outlinks. Tests: - `AbstractFetcherBoltTest` (runs for `FetcherBolt` and `SimpleFetcherBolt`): a 301 with only `SitemapFilter` in the chain, with and without a `Sitemap:` line in robots.txt, for `http.allow.redirects` false and true. The "with sitemap" cases fail on main. - `JSoupParserBoltTest.testHTMLRedirWithSitemapFilter`: meta refresh with `foundSitemap=true`. Fails on main. - New `SitemapFilterTest` for the outlink behaviour that has to stay. Fixes #2282 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
