The GitHub Actions job "Java CI with Maven" on stormcrawler.git/issue-2083-sitemapparser-strict has failed. Run started by GitHub user abhinav-phi (triggered by rzo1).
Head commit for run: d037fa8806c42c9f0f14575ebeb36fa6353dc57e / abhinav-phi <[email protected]> SiteMapParserBolt: strict parsing, opt-in sniffing, non-terminal parse failure (#2083) - the parser is now built in strict mode (sitemap.strict, default true): crawler-commons then discards URLs a sitemap lists on hosts other than its own, so a sitemap cannot enrol URLs on hosts it has nothing to do with, and an HTML page mentioning the sitemap namespace is not parsed leniently into half a sitemap - content sniffing moves behind sitemap.sniffContent (default false, like feed.sniffContent for feeds): a page carrying the namespace string in its first bytes was reclassified as a sitemap, never reached the parser bolt and was never indexed. The key the existing test already set but the bolt ignored is now honoured; a content type which rules a sitemap out (a page served as HTML) stops the sniffing - a document marked as a sitemap through persisted metadata whose body does not parse is emitted as FETCH_ERROR with the isSitemap key dropped, instead of a terminal ERROR: with the archetype's fetchInterval.error of -1, an ERROR removed the URL from the crawl for good, letting whoever controls the content decide what stays in the corpus. On its next fetch the document goes to the parser bolt like any other page The image and all-extensions test sitemaps listed their first URL on www.example.com, which strict mode now correctly excludes; that entry moved under the sitemap's own host. Report URL: https://github.com/apache/stormcrawler/actions/runs/33997599592 With regards, GitHub Actions via GitBox
