abhinav-phi commented on PR #2125: URL: https://github.com/apache/stormcrawler/pull/2125#issuecomment-5641710918
Thanks @dpol1 — pushed 674af288: **Index strict checking:** you're right that crawler-commons' `urlIsValid` only guards `<urlset>` entries and this bolt marks every index `<loc>` as a sitemap. The index loop now applies `SiteMapParser.urlIsValid(indexUrl, target)` when `sitemap.strict` is on, so an index can no longer pull in a sub-sitemap on another host. `crossHostSubSitemapIsNotDiscoveredWhenStrict` pins it, and the yaml comment now says strict covers index `<loc>` entries too. **Redirected sitemap loses isSitemap:** yes — the key is persisted, not transferred to outlinks, so a `/sitemap.xml → /sitemap_index.xml` redirect arrived unmarked and (sniffing off) was stored as a page. Both `FetcherBolt` and `SimpleFetcherBolt` now carry `isSitemap=true` onto the redirect target's outlink when the source was a sitemap. One honest caveat: I could not land a stable fetcher-level regression test for the redirect carry. The fetcher test class shares a fixed WireMock port and the robots parser caches rules per `host:port` statically, so a new async fetch test is order-dependent (it passes in isolation, fails in the class run) — I did not want to ship that flakiness into CI. The change itself is small and verified by the isolated run; happy to add a deterministic test if you can point me at the intended way to reset the robots cache between tests, or I can do it in a follow-up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
