abhinav-phi opened a new pull request, #2125: URL: https://github.com/apache/stormcrawler/pull/2125
Fixes #2083. - **Strict parsing** (`sitemap.strict`, default true): crawler-commons then discards URLs a sitemap lists on hosts other than its own — previously a sitemap could enrol URLs on any host, and those entries skipped `parser.emitOutlinks.max.per.page` and the robots meta tags of the HTML path. - **Opt-in sniffing** (`sitemap.sniffContent`, default false, like `feed.sniffContent` for feeds): any HTML page carrying the sitemaps.org namespace string in its first bytes was reclassified as a sitemap, never reached the parser bolt and was never indexed. The key `SiteMapParserBoltTest` already set but the bolt ignored is now honoured; a content type which rules a sitemap out (a page served as HTML) stops the sniffing when it is enabled. - **Non-terminal parse failure**: a document marked as a sitemap through persisted metadata whose body does not parse is emitted as `FETCH_ERROR` with the `isSitemap` key dropped, instead of a terminal `ERROR`. With the archetype's `fetchInterval.error: -1` an ERROR removed the URL from the crawl for good — in an open crawl that lets a third party remove other people's URLs from the corpus. On its next fetch the document goes to the parser bolt like any other page. **Release note needed:** strict parsing and the sniffing default both change what an existing crawl discovers. Test resources: the image / all-extensions sitemaps listed their first URL on `www.example.com`, which strict mode now correctly excludes; that entry moved under the sitemap's own host. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
