abhinav-phi commented on PR #2125: URL: https://github.com/apache/stormcrawler/pull/2125#issuecomment-5574362411
Thanks — pushed f880c11b closing all three inline points: - **`strict` semantics:** the javadoc, yaml comment and description now say what the flag actually does (strict *URL checking* — a sitemap only yields URLs below its own host and path), not the namespace check. And per your `crawler-default.yaml:290` comment I flipped the default to **false**: the two test fixtures this PR had to rewrite were exactly the warning sign. It stays documented as recommended for open crawls. A new test (`crossSubmittedUrlsAreDiscoveredWithoutStrictChecking`) asserts the default keeps parsing cross-host sitemaps, so both default behaviours are pinned. - **Regression on line 126:** adopted your structure — a document already marked `isSitemap=true` has the namespace sniffed to confirm its type and gets `ct` forced to `application/xml`, so a sitemap served as `text/html` parses again instead of dying with `UnknownFormatException`. Only *promotion* of an unmarked document now requires `sitemap.sniffContent` plus a compatible content type. `sitemapServedAsHtmlStillParses` pins it. With strict off by default, the fixture rewrites (`www.example.com` → the sitemap's own host) are no longer needed for correctness — but they keep the fixtures spec-compliant and the strict-mode test meaningful, so I left them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
