abhinav-phi commented on PR #2125: URL: https://github.com/apache/stormcrawler/pull/2125#issuecomment-5609289260
Thanks — the remaining sniffing bug is fixed in this push (26c7ea59): - sniffsAsSitemap now strips Content-Type parameters before comparing (text/html; profile=xml no longer passes a substring check) and explicitly excludes HTML/XHTML (anything ending in html+xml returns false), so pages are never promoted however much they mention the sitemap namespace. The allow-list is application/xml, text/xml, any +xml, text/plain and application/octet-stream; absent content type still sniffs since the parser guesses anyway. - Tests: sniffingRequiresSitemapCompatibleContentType now actually enables sitemap.sniffContent, plus xhtmlIsNotPromotedToSitemap (application/xhtml+xml), parameterisedHtmlContentTypeIsNotPromoted (text/html; profile=xml) and xmlContentTypeWithNamespaceIsPromotedWhenSniffingEnabled (positive case). sitemapServedAsHtmlStillParses pins the sniff-to-confirm path for declared sitemaps. - The description already states sitemap.strict defaults to false (fixed with the defaults change). mvn -pl core verify -DCI_ENV=true is green on the branch. Please re-review when you get a moment. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
